Polarization Image Fusion with Self-Learned Fusion Strategy

Junchao Zhang,Jianbo Shao,Jianlai Chen,Degui Yang,Buge Liang

DOI: https://doi.org/10.1016/j.patcog.2021.108045

IF: 8

2021-01-01

Pattern Recognition

Abstract:•A new unsupervised fusion network by integrating a Fusion Sub-Net is proposed.•A novel loss function is performed to address fusion problems.•The trained network on polarization data works well on other fusion tasks.

What problem does this paper attempt to address?

Polarization image fusion method with image enhancement

Chenguang Wang,Ruyue Ma,Deli Yan,Huiliang Cao,Chong Shen

DOI: https://doi.org/10.1088/1402-4896/ad1a2f

2024-01-18

Physica Scripta

Abstract:Abstract Because of their complementary characteristics, intensity images and polarization images are often fused to produce information-rich images. However, the polarization characteristics are easily affected by the object’s environment, and the image fusion process may lose important information. In this paper, we propose an unsupervised end-to-end network framework based on a CNN for intensity images and degree of linear polarization images. First, we construct our own polarization dataset to solve the limitations of the training dataset; a hybrid loss function is designed to form an unsupervised learning process; and a Laplace operator enhancement layer is introduced into the network to further improve the quality of the fused images. Subjective and objective comparison experiments prove that the proposed fusion network is visually superior to several classical fusion methods.

physics, multidisciplinary
Learning based polarization image fusion under an alternative paradigm

Jianwen Meng,Wenyi Ren,Ruoning Yu,Xu Ma,Gonzalo R. Arce,Dan Wu,Rui Zhang,Yingge Xie

DOI: https://doi.org/10.1016/j.optlastec.2023.109969

2023-09-01

Abstract:Polarization image fusion is a crucial component of polarization imaging applications. Most of the existing polarization fusion algorithms concentrate on fusing the intensity and the degree of linear polarization (DoLP). The information encoded in the angle of linear polarization (AoLP), such as surface orientation and illumination, is not introduced in existing fusion frameworks due to the noise-sensitive property, the π -ambiguity and diffuse/specular-ambiguity. To address this problem, we adopt a new polarization mapping paradigm as an alternative to improve feature utilization and information interpretability. A learning based polarization image fusion network is proposed to learn the potential features and recreate the intuitively understandable images. Four public polarization datasets are introduced in the experiments. The linear polarization information was effectively fused by the proposed method. The noise and distortion introduced by DoLP and AoLP are suppressed meanwhile. According to the evaluation and analysis, it found that the fused images acquired by the proposed method outperform the state-of-the-art methods in the aspects of target surface orientation representation, low-illumination object recognition, and texture enhancement.

optics,physics, applied
A Self-Supervised Spaceborne Multispectral and Hyperspectral Image Fusion Unrolling Network

Zengliang Zhu,Xinyu Wang,Guanzhong Li,Yanfei Zhong

DOI: https://doi.org/10.1109/tgrs.2024.3413023

IF: 8.2

2024-01-01

IEEE Transactions on Geoscience and Remote Sensing

Abstract:Deep learning has emerged as the predominant approach for multispectral and hyperspectral image fusion. However, most fusion networks are typically trained and validated on hyperspectral and multispectral image pairs generated from the same hyperspectral images, with degradation simulations inconsistent with real situations and relatively limited volumes of images. When transferring a pretrained multispectral and hyperspectral image fusion model from ground or airborne images to spaceborne images, it encounters a larger dataset and more complex spatial-spectral degradation, leading to spectral distortions and spatial artifacts in the fused images. In this article, the challenges associated with the transfer are addressed through the introduction of a self-supervised multispectral and hyperspectral image fusion unrolling network for spaceborne imagery, termed as MH-FUNet. MH-FUNet adopts a self-supervised paradigm to learn a robust mapping from spaceborne data. It utilizes a deep unrolling network to iteratively refine fusion results from coarse to fine. To account for spatial scale differences between the self-supervised training and test datasets, a multiscale fusion strategy is introduced. This strategy is combined with spectral and spatial attention mechanisms to restore spatial and spectral details. Additionally, a gradient constraint unit is proposed to maintain spatial consistency when up-scaling low-resolution hyperspectral imagery. Performance evaluation of the proposed method is conducted against state-of-the-art fusion techniques on both simulated Chikusei dataset and the proposed real WHU-MHF dataset, which consists of simultaneously observed hyperspectral and multispectral image pairs. MH-FUNet outperforms existing methods across all datasets, demonstrating superior performance in spaceborne multispectral and hyperspectral image fusion experiments.
Polarized Prior Guided Fusion Network for Infrared Polarization Images

Kunyuan Li,Meibin Qi,Shuo Zhuang,Yimin Liu

DOI: https://doi.org/10.1109/tgrs.2024.3389976

IF: 8.2

2024-04-27

IEEE Transactions on Geoscience and Remote Sensing

Abstract:Typical infrared polarization image fusion aims to integrate background details in the infrared intensity and salient target in the degree of linear polarization (DoLP). Many fusion methods show advanced network architecture, but few works can form effective feature representations for the differences in prior distributions of the infrared intensity and DoLP, and the interference of DoLP with noise makes fusion more challenging. This article uses a learned low-rank decomposition model to extract low-rank representations containing background details in infrared intensity and sparse features with salient targets in DoLP. To reduce noise interference, we design a fusion module based on an attention-guided filter, where the infrared intensity serves as a guide map to suppress the background in DoLP. Moreover, a novel loss constraint is proposed to improve the fusion performance. Specifically, the fusion network is trained by reconstructing polarized images in different directions from the fused image. Quantitative and qualitative experimental results validate the effectiveness of our approach. In comparison to existing methods, our fusion model can better preserve the polarization salient target and suppress the background interference with fewer parameters. The source code is available at https://github.com/lkyahpu/PIPFNet.

imaging science & photographic technology,remote sensing,engineering, electrical & electronic,geochemistry & geophysics
NLFNet: Non-Local Fusion Towards Generalized Multimodal Semantic Segmentation Across RGB-Depth, Polarization, and Thermal Images

Ran Yan,Kailun Yang,Kaiwei Wang

DOI: https://doi.org/10.1109/robio54168.2021.9739390

2021-01-01

Abstract:In recent years, intelligent driving navigation has made considerable progress, and semantic segmentation is one of the most advanced scene perception methods. At present, traditional semantic segmentation methods can use RGB images for detection of obstacles that are clearly visible in outdoor scenes. However, in the face of complex realistic driving scenes, RGB images cannot provide sufficient information. We need some other modal information to supplement the RGB information. In this paper, we propose Non-Local Fusion Network (NLFNet), which is a semantic segmentation network that can selectively fuse multimodal input information in an adaptive manner. It can use complementary information collected by different optical sensors to extract effective features for fusion. Thereby, it improves the segmentation accuracy of the network and solves the problem of object recognition in various challenging real-world scenes. We conduct comprehensive experiments to verify the effectiveness and generalization ability of the framework across RGB-Depth, RGB-Polarization, and RGB-Thermal image semantic segmentation, which is especially suitable for autonomous driving and robot vision applications.
Noise-aware Infrared Polarization Image Fusion Based on Salient Prior with Attention-Guided Filtering Network.

Kunyuan Li,Meibin Qi,Shuo Zhuang,Yimin Liu,Jun Gao

DOI: https://doi.org/10.1364/oe.492954

IF: 3.8

2023-01-01

Optics Express

Abstract:Infrared polarization image fusion integrates intensity and polarization information, producing a fused image that enhances visibility and captures crucial details. However, in complex environments, polarization imaging is susceptible to noise interference. Existing fusion methods typically use the infrared intensity (S0) and degree of linear polarization (DoLP) images for fusion but fail to consider the noise interference, leading to reduced performance. To cope with this problem, we propose a fusion method based on polarization salient prior, which extends DoLP by angle of polarization (AoP) and introduces polarization distance (PD) to obtain salient target features. Moreover, according to the distribution difference between S0 and DoLP features, we construct a fusion network based on attention-guided filtering, utilizing cross-attention to generate filter kernels for fusion. The quantitative and qualitative experimental results validate the effectiveness of our approach. Compared with other fusion methods, our method can effectively suppress noise interference and preserve salient target features.
Unsupervised end-to-end infrared and visible image fusion network using learnable fusion strategy

Yili Chen,Minjie Wan,Yunkai Xu,Xiqing Cao,Xiaojie Zhang,Qian Chen,Guohua Gu

DOI: https://doi.org/10.1364/JOSAA.473908

2022-12-01

Abstract:Infrared and visible image fusion aims to reconstruct fused images with comprehensive visual information by merging the complementary features of source images captured by different imaging sensors. This technology has been widely used in civil and military fields, such as urban security monitoring, remote sensing measurement, and battlefield reconnaissance. However, the existing methods still suffer from the preset fusion strategies that cannot be adjustable to different fusion demands and the loss of information during the feature propagation process, thereby leading to the poor generalization ability and limited fusion performance. Therefore, we propose an unsupervised end-to-end network with learnable fusion strategy for infrared and visible image fusion in this paper. The presented network mainly consists of three parts, including the feature extraction module, the fusion strategy module, and the image reconstruction module. First, in order to preserve more information during the process of feature propagation, dense connections and residual connections are applied to the feature extraction module and the image reconstruction module, respectively. Second, a new convolutional neural network is designed to adaptively learn the fusion strategy, which is able to enhance the generalization ability of our algorithm. Third, due to the lack of ground truth in fusion tasks, a loss function that consists of saliency loss and detail loss is exploited to guide the training direction and balance the retention of different types of information. Finally, the experimental results verify that the proposed algorithm delivers competitive performance when compared with several state-of-the-art algorithms in terms of both subjective and objective evaluations. Our codes are available at https://github.com/MinjieWan/Unsupervised-end-to-end-infrared-and-visible-image-fusion-network-using-learnable-fusion-strategy.
Polarization Image Fusion Algorithm Based on Improved PCNN

Siyuan Zhang,Yan Yuan,Lijuan Su,Liang Hu,Hui Liu

DOI: https://doi.org/10.1117/12.2037173

2013-01-01

Abstract:The polarization detection technique provides polarization information of objects which conventional detection techniques are unable to obtain. In order to fully utilize of obtained polarization information, various polarization imagery fusion algorithms have been developed. In this research, we proposed a polarization image fusion algorithm based on the improved pulse coupled neural network (PCNN). The improved PCNN algorithm uses polarization parameter images to generate the fused polarization image with object details for polarization information analysis and uses the matching degree M as the fusion rule. The improved PCNN fused image is compared with fused images based on Laplacian pyramid (LP) algorithm, Wavelet algorithm and PCNN algorithm. Several performance indicators are introduced to evaluate the fused images. The comparison showed the presented algorithm yields image with much higher quality and preserves more detail information of the objects.
Joint target geometry and polarization properties for polarization image fusion

Jin Duan,Ju Liu,Youfei Hao,Guangqiu Chen,Yue Zheng,Linfeng Jia

DOI: https://doi.org/10.1016/j.optlaseng.2024.108176

IF: 5.666

2024-03-22

Optics and Lasers in Engineering

Abstract:Traditional polarization image fusion focuses on mining information from the source image and realizes polarization image fusion by finding the feature balance point between two source images. However, this method is strictly limited to high quality and sufficiently informative source polarization images. In addition, if the fusion rules are formulated in terms of the dimensions of the source image information, it is difficult to achieve accurate and efficient fusion results. Based on this, this paper designs a new polarization image fusion model that joins the target geometry and material polarization characteristics from the starting polarization characteristics of different materials (GM-PFNet). In this implementation, a weak reference polarization image quality assessment method (WR-PIQA) for joint target geometry and material polarization characteristics is designed by exploring the imaging laws of scene target geometry and material polarization characteristics. This method utilizes the polarization parameters of target surface roughness, specular reflection, and diffuse reflection, distinguishing it from traditional feature extraction methods. It employs image quality assessment weighting to explore new information from the source images. Since the model belongs to a data-driven self-evolving training model, it is able to utilize the obtained intermediate fusion results to further co-supervise the fused images during the training process. In this way, our fusion results can benefit both from learning from the original input image and from the intermediate output of the network itself. A comparison with state-of-the-art methods on both self-constructed and publicly available datasets reveals that our GM-PFNet model achieves superior performance in both qualitative and quantitative experiments.

optics
UNIFusion: A Lightweight Unified Image Fusion Network

Chunyang Cheng,Xiao-Jun Wu,Tianyang Xu,Guoyang Chen

DOI: https://doi.org/10.1109/tim.2021.3109379

IF: 5.6

2021-01-01

IEEE Transactions on Instrumentation and Measurement

Abstract:Advanced image fusion frameworks have focused on constructing a unified system to deliver general fusion solutions for all the potential subtasks, including visible and infrared image fusion, multi-focus image fusion, and multi-exposure image fusion. However, the fusion performance of these unified solutions cannot be guaranteed as the involved subtasks exhibit essential diversity. Besides, the huge volume of these models also sacrifices their flexibility in practical applications. To this end, we propose a lightweight unified fusion network to balance the multi-level information featured across different channels and different layers. In particular, We construct a novel network architecture based on the Ghost module endowed with a new loss function. The depth of the designed network is increased, while the number of parameters is reduced by an order of magnitude compared to existing learning-based fusion approaches. In principle, our network belongs to the autoencoder paradigm, comprising three parts: encoder, fusion layer, and decoder, respectively. Considering that the efficacy of existing autoencoder-based approaches is impeded by the single fusion strategy, we use the guided filtering approach to decompose the source image into a base layer and a detail layer to extend the diversity of the inputs. Drawing on this, we can design different fusion strategies in these two layers for adapting different image fusion tasks. In the fusion layer, besides the three basic fusion strategies, i.e., average, max, and spatial attention, an additional gradient perception strategy is proposed to handle the multi-focus image fusion problem, improving the corresponding fusion performance. Qualitative and quantitative experiments show that our UNIFusion achieves promising results in these image fusion tasks, compared to the state of the art.

engineering, electrical & electronic,instruments & instrumentation
Infrared and Visible Image Fusion Based on a Two-Stage Class Conditioned Auto-Encoder Network.

Yanpeng Cao,Xing Luo,Xi Tong,Jiangxin Yang,Yanlong Cao

DOI: https://doi.org/10.1016/j.neucom.2023.126248

IF: 6

2023-01-01

Neurocomputing

Abstract:Existing auto-encoder based infrared and visible image fusion methods typically utilize a shared encoder to extract features from different modalities and adopt a handcrafted fusion strategy to fuse the extracted features into intermediate representation before the decoder part. In this paper, we present a novel two-stage class conditioned auto-encoder framework for high-quality multispectral fusion tasks. In the first training stage, we introduce a class embedding sub-branch to the encoder network for modeling the characteristics of different modalities and adaptively scaling the intermediate features based on the input modality. Moreover, we design a cross-transfer residual block to promote the content and texture information flow in the encoder for generating more representative features. In the second training stage, we insert a learnable fusion module between the pre-trained class conditioned encoder and decoder parts to replace the handcrafted fusion strategy. Specific intensity and gradient loss functions are utilized to tune the model for the fusion of distinctive deep features in a data-driven manner. With the important designs including the class conditioned auto-encoder and the two-stage training strategy, our proposed TS-ClassFuse can better preserve distinctive information/features from the source images and decrease the training difficulty for simultaneously extracting informative features and determining the optimal fusion scheme. Experimental results verify the effectiveness of our method in terms of both qualitative and quantitative evaluations.
Underwater Target Detection Utilizing Polarization Image Fusion Algorithm Based on Unsupervised Learning and Attention Mechanism

Haoyuan Cheng,Deqing Zhang,Jinchi Zhu,Hao Yu,Jinkui Chu

DOI: https://doi.org/10.3390/s23125594

IF: 3.9

2023-06-16

Sensors

Abstract:Since light propagation in water bodies is subject to absorption and scattering effects, underwater images using only conventional intensity cameras will suffer from low brightness, blurred images, and loss of details. In this paper, a deep fusion network is applied to underwater polarization images; that is, the underwater polarization images are fused with intensity images using the deep learning method. To construct a training dataset, we establish an experimental setup to obtain underwater polarization images and perform appropriate transformations to expand the dataset. Next, an end-to-end learning framework based on unsupervised learning and guided by an attention mechanism is constructed for fusing polarization and light intensity images. The loss function and weight parameters are elaborated. The produced dataset is used to train the network under different loss weight parameters, and the fused images are evaluated based on different image evaluation metrics. The results show that the fused underwater images are more detailed. Compared with light intensity images, the information entropy and standard deviation of the proposed method increase by 24.48% and 139%. The image processing results are better than other fusion-based methods. In addition, the improved U-net network structure is used to extract features for image segmentation. The results show that the target segmentation based on the proposed method is feasible under turbid water. The proposed method does not require manual adjustment of weight parameters, has faster operation speed, and has strong robustness and self-adaptability, which is important for research in vision fields, such as ocean detection and underwater target recognition.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
A dual-weighted polarization image fusion method based on quality assessment and attention mechanisms

Jin Duan,Hao Zhang,Ju Liu,Meiling Gao,Cai Cheng,Guangqiu Chen

DOI: https://doi.org/10.3389/fphy.2023.1214206

IF: 3.718

2023-07-05

Frontiers in Physics

Abstract:This paper proposes a dual-weighted polarization image fusion method based on quality assessment and attention mechanisms to fuse the intensity image (S0) and the degree of linear polarization (DoLP). S0 has high contrast and clear details, and DoLP has an outstanding ability to characterize polarization properties, so the fusion can achieve an effective complementation of superior information. We decompose S0 and DoLP into base layers and detail layers. In the base layers, we build a quality assessment unit combining information entropy, no-reference image quality assessment, and local energy to ensure the fused image has high contrast and clear and natural visual perception; in the detail layer, we first extract depth features using the pre-trained VGG19, then construct an attention enhancement unit combining space and channels, and finally effectively improve the preservation of detail information and edge contours in the fused image. The proposed method is able to perceive and retain polarization image features sufficiently to obtain desirable fusion results. Comparing nine typical fusion methods on two publicly available and own polarization datasets, experimental results show that the proposed method outperforms other comparative algorithms in both qualitative comparison and quantitative analysis.

physics, multidisciplinary
MUFusion: A general unsupervised image fusion network based on memory unit

Chunyang Cheng,Tianyang Xu,Xiao-Jun Wu

DOI: https://doi.org/10.1016/j.inffus.2022.11.010

IF: 18.6

2022-11-15

Information Fusion

Abstract:Existing image fusion approaches are committed to using a single deep network to solve different image fusion problems, achieving promising performance in recent years. However, devoid of the ground-truth output, in these methods, only the appearance from source images can be exploited during the training process to generate the fused images, resulting in suboptimal solutions. To this end, we advocate a self-evolutionary training formula by introducing a novel memory unit architecture (MUFusion). In this unit, specifically, we utilize the intermediate fusion results obtained during the training process to further collaboratively supervise the fused image. In this way, our fusion results can not only learn from the original input images, but also benefit from the intermediate output of the network itself. Furthermore, an adaptive unified loss function is designed based on this memory unit, which is composed of two loss items, i.e. , content loss and memory loss. In particular, the content loss is calculated based on the activity level maps of source images, which can constrain the output image to contain specific information. On the other hand, the memory loss is obtained based on the previous output of our model, which is utilized to force the network to yield fusion results with higher quality. Considering the handcrafted activity level maps cannot consistently reflect the accurate salience judgement, we put two adaptive weight items between them to prevent this degradation phenomenon. In general, our MUFusion can effectively handle a series of image fusion tasks, including infrared and visible image fusion, multi-focus image fusion, multi-exposure image fusion, and medical image fusion. Particularly, the source images are concatenated in the channel dimension. After that, a densely connected feature extraction network with two scales is used to extract the deep features of the source images. Following this, the fusion result is obtained by two feature reconstruction blocks with skip connections from the feature extraction network. Qualitative and quantitative experiments on 4 image fusion subtasks demonstrate the superiority of our MUFusion, compared to the state-of-the-art methods.

computer science, artificial intelligence, theory & methods
Polarization Fusion Algorithm Based on NSCT Decomposition and Improved Dual-Channel PCNN Model

Chao Hu,Bin Fan,Jiang Bian,Shuo Zhong,Wang Lun,Mengxia Hou,Bo Qi

DOI: https://doi.org/10.1117/12.3021457

2024-01-01

Abstract:Polarization imaging is capable of effectively showcasing the polarization properties of objects while also mitigating the impact of strong light and enhancing the visibility of weak light. As a result, it compensates for the limitations of intensity imaging in low-light environments. To fully utilize the advantages of intensity and polarization images, this paper proposes a method that combines NSCT decomposition and an improved dual-channel PCNN model. The method enhances the traditional PCNN model by extending the input to dual channels, employing adaptively calculated parameters, and applying adaptive linking weights to the high- and low-frequency images obtained from NSCT decomposition. The highfrequency image utilizes the MSMG operator, while the low-frequency image employs a weight map controlled by multiple parameters. This method is utilized for fusion processing of intensity and polarization images.By conducting comparative experiments with six commonly used algorithms, the results demonstrate the superior performance of this method in terms of preserving texture details, providing high-quality images, achieving information-rich fusion, generating fused images with rich grayscale levels, and maintaining structural similarity. In summary, the proposed method exhibits significant advantages in fusing intensity and polarization images.
Contrast Enhanced Color Polarization Image Fusion

Jianwen Meng,Wenyi Ren,Ruoning Yu,Dan Wu,Rui Zhang,Yingge Xie,Jian Wang

DOI: https://doi.org/10.1016/j.ijleo.2023.170935

IF: 3.1

2023-01-01

Optik

Abstract:Polarization image fusion is commonly employed to improve the quality of images in various fields such as object recognition and detection. There have been limited investigations on color-based polarization image fusion methods, which predominantly rely on pseudo-color approaches and are primarily suited for human perception. We propose a contrast-enhanced method for color polarization image fusion based on local contrast and multiscale decomposition to effectively reflect the multidimensional information and assist subsequent vision tasks. The method created a basic local contrast map from the luminance channel of intensity and grayscale map of the degree of linear polarization (DoLP), then extracted high-frequency features to fuse with the local contrast map utilizing rolling fast guided filtering and least squares. Finally, the local contrast map and high-frequency features of DoLP were combined and injected into the color-intensity image. The quantitative and qualitative evaluation based on nine color image fusion methods and eight blind image assessment measures revealed that the proposed algorithm not only achieved image contrast improvement and effective information fusion, but also preserved the color information.
Fully Polarimetric SAR and Single-Polarization SAR Image Fusion Network

Liupeng Lin,Jie Li,Huanfeng Shen,Lingli Zhao,Qiangqiang Yuan,Xinghua Li

DOI: https://doi.org/10.48550/arXiv.2107.08355

2021-07-18

Image and Video Processing

Abstract:The data fusion technology aims to aggregate the characteristics of different data and obtain products with multiple data advantages. To solves the problem of reduced resolution of PolSAR images due to system limitations, we propose a fully polarimetric synthetic aperture radar (PolSAR) images and single-polarization synthetic aperture radar SAR (SinSAR) images fusion network to generate high-resolution PolSAR (HR-PolSAR) images. To take advantage of the polarimetric information of the low-resolution PolSAR (LR-PolSAR) image and the spatial information of the high-resolution single-polarization SAR (HR-SinSAR) image, we propose a fusion framework for joint LR-PolSAR image and HR-SinSAR image and design a cross-attention mechanism to extract features from the joint input data. Besides, based on the physical imaging mechanism, we designed the PolSAR polarimetric loss function for constrained network training. The experimental results confirm the superiority of fusion network over traditional algorithms. The average PSNR is increased by more than 3.6db, and the average MAE is reduced to less than 0.07. Experiments on polarimetric decomposition and polarimetric signature show that it maintains polarimetric information well.
FDFNet: A Fusion Network for Generating High-Resolution Fully PolSAR Images

Liupeng Lin,Huanfeng Shen,Jie Li,Qiangqiang Yuan

DOI: https://doi.org/10.1109/lgrs.2021.3127958

IF: 5.343

2022-01-01

IEEE Geoscience and Remote Sensing Letters

Abstract:Deep learning shows potential superiority in the image fusion field. To solve the problem of the spatial resolution degradation of polarimetric synthetic aperture radar (PolSAR) images caused by system limitation, we propose a fully PolSAR images and DualSAR images fusion network (FDFNet). We use low resolution (LR)-PolSAR super-resolution (LPSR) and modified cross attention mechanism (MCroAM) to perform data fusion on LR-PolSAR and high resolution (HR)-dual-polarization synthetic aperture radar (DualSAR) and design a polarimetric decomposition attention module to introduce the polarimetric parameters of LR-PolSAR images to maintain polarimetric information. Besides, we use the differential information between LR-PolSAR and HR-DualSAR to guide spatial resolution reconstruction. The loss function based on the $L_{1} $ norm is used to constrain the network training process. The experimental results show the superiority of the proposed method over the existing methods in visual and quantitative evaluation. In addition, polarimetric decomposition experiments verify the effectiveness of the proposed method to maintain polarimetric information.

imaging science & photographic technology,remote sensing,engineering, electrical & electronic,geochemistry & geophysics
Fusion from Decomposition: A Self-Supervised Approach for Image Fusion and Beyond

Pengwei Liang,Junjun Jiang,Qing Ma,Xianming Liu,Jiayi Ma

2024-10-16

Abstract:Image fusion is famous as an alternative solution to generate one high-quality image from multiple images in addition to image restoration from a single degraded image. The essence of image fusion is to integrate complementary information from source images. Existing fusion methods struggle with generalization across various tasks and often require labor-intensive designs, in which it is difficult to identify and extract useful information from source images due to the diverse requirements of each fusion task. Additionally, these methods develop highly specialized features for different downstream applications, hindering the adaptation to new and diverse downstream tasks. To address these limitations, we introduce DeFusion++, a novel framework that leverages self-supervised learning (SSL) to enhance the versatility of feature representation for different image fusion tasks. DeFusion++ captures the image fusion task-friendly representations from large-scale data in a self-supervised way, overcoming the constraints of limited fusion datasets. Specifically, we introduce two innovative pretext tasks: common and unique decomposition (CUD) and masked feature modeling (MFM). CUD decomposes source images into abstract common and unique components, while MFM refines these components into robust fused features. Jointly training of these tasks enables DeFusion++ to produce adaptable representations that can effectively extract useful information from various source images, regardless of the fusion task. The resulting fused representations are also highly adaptable for a wide range of downstream tasks, including image segmentation and object detection. DeFusion++ stands out by producing versatile fused representations that can enhance both the quality of image fusion and the effectiveness of downstream high-level vision tasks, simplifying the process with the elegant fusion framework.

Computer Vision and Pattern Recognition
U2Fusion: A Unified Unsupervised Image Fusion Network

Han Xu,Jiayi Ma,Junjun Jiang,Xiaojie Guo,Haibin Ling

DOI: https://doi.org/10.1109/tpami.2020.3012548

IF: 23.6

2022-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:This study proposes a novel unified and unsupervised end-to-end image fusion network, termed as U2Fusion, which is capable of solving different fusion problems, including multi-modal, multi-exposure, and multi-focus cases. Using feature extraction and information measurement, U2Fusion automatically estimates the importance of corresponding source images and comes up with adaptive information preservation degrees. Hence, different fusion tasks are unified in the same framework. Based on the adaptive degrees, a network is trained to preserve the adaptive similarity between the fusion result and source images. Therefore, the stumbling blocks in applying deep learning for image fusion, e.g., the requirement of ground-truth and specifically designed metrics, are greatly mitigated. By avoiding the loss of previous fusion capabilities when training a single model for different tasks sequentially, we obtain a unified model that is applicable to multiple fusion tasks. Moreover, a new aligned infrared and visible image dataset, RoadScene (available at https://github.com/hanna-xu/RoadScene), is released to provide a new option for benchmark evaluation. Qualitative and quantitative experimental results on three typical image fusion tasks validate the effectiveness and universality of U2Fusion. Our code is publicly available at https://github.com/hanna-xu/U2Fusion.

computer science, artificial intelligence,engineering, electrical & electronic

Polarization Image Fusion with Self-Learned Fusion Strategy

Polarization image fusion method with image enhancement

Learning based polarization image fusion under an alternative paradigm

A Self-Supervised Spaceborne Multispectral and Hyperspectral Image Fusion Unrolling Network

Polarized Prior Guided Fusion Network for Infrared Polarization Images

NLFNet: Non-Local Fusion Towards Generalized Multimodal Semantic Segmentation Across RGB-Depth, Polarization, and Thermal Images

Noise-aware Infrared Polarization Image Fusion Based on Salient Prior with Attention-Guided Filtering Network.

Unsupervised end-to-end infrared and visible image fusion network using learnable fusion strategy

Polarization Image Fusion Algorithm Based on Improved PCNN

Joint target geometry and polarization properties for polarization image fusion

UNIFusion: A Lightweight Unified Image Fusion Network

Infrared and Visible Image Fusion Based on a Two-Stage Class Conditioned Auto-Encoder Network.

Underwater Target Detection Utilizing Polarization Image Fusion Algorithm Based on Unsupervised Learning and Attention Mechanism

A dual-weighted polarization image fusion method based on quality assessment and attention mechanisms

MUFusion: A general unsupervised image fusion network based on memory unit

Polarization Fusion Algorithm Based on NSCT Decomposition and Improved Dual-Channel PCNN Model

Contrast Enhanced Color Polarization Image Fusion

Fully Polarimetric SAR and Single-Polarization SAR Image Fusion Network

FDFNet: A Fusion Network for Generating High-Resolution Fully PolSAR Images

Fusion from Decomposition: A Self-Supervised Approach for Image Fusion and Beyond

U2Fusion: A Unified Unsupervised Image Fusion Network