The Milky Way: From Mammary Gland to Milk to Newborn

M. Hamosh

DOI: https://doi.org/10.1007/978-1-4615-0559-4_2

Abstract:

What problem does this paper attempt to address?

Fusion of Low-Illuminance Visible and Near-Infrared Images Based on Convolutional Neural Networks

Tang Chaoying,Pu Shiliang,Ye Pengzhao,Xiao Fei,Feng Huajun

DOI: https://doi.org/10.3788/aos202040.1610001

2020-01-01

Abstract:Herein, for low-illumination application scenes, an end-to-end convolutional neural network (CNN) is proposed for the fusion of near-infrared (NIR) and visible images. The fused image can combine the signal-to-noise ratio of an NIR image and the color of visible images. To verify the capability of CNN for practical fusion tasks, a real dataset with accurate registration was collected. Moreover, the training set was preprocessed via information fusion, thereby enabling the network to extract additional information from NIR images. Experimental results reveal that the proposed method is superior to existing fusion methods in terms of visual quality and quantitative measurements.
Infrared and Visible Image Fusion Based on a Two-Stage Class Conditioned Auto-Encoder Network.

Yanpeng Cao,Xing Luo,Xi Tong,Jiangxin Yang,Yanlong Cao

DOI: https://doi.org/10.1016/j.neucom.2023.126248

IF: 6

2023-01-01

Neurocomputing

Abstract:Existing auto-encoder based infrared and visible image fusion methods typically utilize a shared encoder to extract features from different modalities and adopt a handcrafted fusion strategy to fuse the extracted features into intermediate representation before the decoder part. In this paper, we present a novel two-stage class conditioned auto-encoder framework for high-quality multispectral fusion tasks. In the first training stage, we introduce a class embedding sub-branch to the encoder network for modeling the characteristics of different modalities and adaptively scaling the intermediate features based on the input modality. Moreover, we design a cross-transfer residual block to promote the content and texture information flow in the encoder for generating more representative features. In the second training stage, we insert a learnable fusion module between the pre-trained class conditioned encoder and decoder parts to replace the handcrafted fusion strategy. Specific intensity and gradient loss functions are utilized to tune the model for the fusion of distinctive deep features in a data-driven manner. With the important designs including the class conditioned auto-encoder and the two-stage training strategy, our proposed TS-ClassFuse can better preserve distinctive information/features from the source images and decrease the training difficulty for simultaneously extracting informative features and determining the optimal fusion scheme. Experimental results verify the effectiveness of our method in terms of both qualitative and quantitative evaluations.
Fusion of Infrared and Visible Images Via Multi-Layer Convolutional Sparse Representation

Zhouyu Zhang,Chenyuan He,Hai Wang,Yingfeng Cai,Long Chen,Zhihua Gan,Fenghua Huang,Yiqun Zhang

DOI: https://doi.org/10.1016/j.jksuci.2024.102090

IF: 9.006

2024-01-01

Journal of King Saud University - Computer and Information Sciences

Abstract:Infrared and visible image fusion is an effective solution for image quality enhancement. However, conventional fusion models require the decomposition of source images into image blocks, which disrupts the original structure of the images, leading to the loss of detail in the fused images and making the fusion results highly sensitive to matching errors. This paper employs Convolutional Sparse Representation (CSR) to perform global feature transformation on the source images, overcoming the drawbacks of traditional fusion models that rely on image decomposition. Inspired by neural networks, a multi-layer CSR model is proposed, which involves five layers in a forward-feeding manner: two CSR layers acquiring sparse coefficient maps, one fusion layer combining sparse maps, and two reconstruction layers for image recovery. The dataset used in this paper comprises infrared and visible images selected from public dataset, as well as registered images collected by an actual Unmanned Aerial Vehicle (UAV). The source images contain ground targets, marine targets, and natural landscapes. To validate the effectiveness of the proposed image fusion model in this paper, comparative analysis is conducted with state-of-the-art (SOTA) algorithms. Experimental results demonstrate that the proposed fusion model outperforms other state-of-the-art methods by at least 10% in SF, EN, MI and QAB/F fusion metrics in most image fusion cases, thereby affirming its favorable performance.
Infrared and visible image fusion with entropy-based adaptive fusion module and mask-guided convolutional neural network

Jianming Zhang,Wenxin Lei,Shuyang Li,Zongping Li,Xudong Li

DOI: https://doi.org/10.1016/j.infrared.2023.104629

2023-03-04

Abstract:Since infrared and visible images have different descriptions of the same scene and have a lot of complementary information, they are usually fused to obtain a more comprehensive image. To overcome the detail information losing, fusion artifacts, and unnatural fusion in the traditional image fusion, we propose a novel algorithm for infrared and visible image fusion. Firstly, we introduce a guided filter to decompose the input image into a first-level base layer, a second-level base layer and a detail layer. Specifically, for the first-level base layer fusion, we propose an entropy-based image adaptive fusion module, which is guided by the image quality and the information entropy. To reduce the low-frequency information losing, we use the maximum absolute value rule to fuse the second-level base layer. In addition, for the detail layer containing many textures and edges, we construct a mask-guided deep convolutional neural network to fuse it. We label the infrared image pixel-by-pixel in the training set to obtain the foreground target mask, and construct the loss function jointly considering the mask, gradient, and the structural similarity of the image. Masks are only used when training the network. Finally, we reconstruct the image. Experiments show that our algorithm has achieved good performance in both subjective and objective evaluation.

optics,physics, applied,instruments & instrumentation
EV-Fusion: A Novel Infrared and Low-Light Color Visible Image Fusion Network Integrating Unsupervised Visible Image Enhancement

Xin Zhang,Xia Wang,Changda Yan,Qiyang Sun

DOI: https://doi.org/10.1109/jsen.2023.3346886

IF: 4.3

2024-02-16

IEEE Sensors Journal

Abstract:Infrared and visible image fusion can effectively integrate the advantages of two source images, preserving significant target information and rich texture details. However, most existing fusion methods are only designed for well-illuminated scenes and tend to lose details when encountering low-light scenes because of the poor brightness of visible images. Some methods incorporate a light adjustment module, but they typically focus only on enhancing intensity information and neglect the enhancement of color feature, resulting in unsatisfactory visual effects in the fused images. To address this issue, this article proposes a novel method called EV-fusion, which explores the potential color and detail features in visible images and improve the visual perception of fused images. Specifically, an unsupervised image enhancement module is designed that effectively restores texture, structure, and color information in visible images by several non-reference loss functions. Then, an intensity image fusion module is devised to integrate the enhanced visible image and the infrared image. Moreover, to improve the infrared salient object feature in the fused images, we propose an infrared bilateral-guided salience map embedding into the fusion loss functions. Extensive experiments demonstrate that our method outperforms state-of-the-art (SOTA) infrared visible image fusion methods.

engineering, electrical & electronic,instruments & instrumentation,physics, applied
A Multi-Stage Visible and Infrared Image Fusion Network Based on Attention Mechanism

Xin Zheng,Qiyong Yang,Pengbo Si,Qiang Wu

DOI: https://doi.org/10.3390/s22103651

IF: 3.9

2022-05-12

Sensors

Abstract:Pixel-level image fusion is an effective way to fully exploit the rich texture information of visible images and the salient target characteristics of infrared images. With the development of deep learning technology in recent years, the image fusion algorithm based on this method has also achieved great success. However, owing to the lack of sufficient and reliable paired data and a nonexistent ideal fusion result as supervision, it is difficult to design a precise network training mode. Moreover, the manual fusion strategy has difficulty ensuring the full use of information, which easily causes redundancy and omittance. To solve the above problems, this paper proposes a multi-stage visible and infrared image fusion network based on an attention mechanism (MSFAM). Our method stabilizes the training process through multi-stage training and enhances features by the learning attention fusion block. To improve the network effect, we further design a Semantic Constraint module and Push–Pull loss function for the fusion task. Compared with several recently used methods, the qualitative comparison intuitively shows more beautiful and natural fusion results by our model with a stronger applicability. For quantitative experiments, MSFAM achieves the best results in three of the six frequently used metrics in fusion tasks, while other methods only obtain good scores on a single metric or a few metrics. Besides, a commonly used high-level semantic task, i.e., object detection, is used to prove its greater benefits for downstream tasks compared with singlelight images and fusion results by existing methods. All these experiments prove the superiority and effectiveness of our algorithm.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
Infrared and Visible Image Fusion with Convolutional Neural Networks.

Yu Liu,Xun Chen,Juan Cheng,Hu Peng,Zengfu Wang

DOI: https://doi.org/10.1142/s0219691318500182

2018-01-01

Abstract:The fusion of infrared and visible images of the same scene aims to generate a composite image which can provide a more comprehensive description of the scene. In this paper, we propose an infrared and visible image fusion method based on convolutional neural networks (CNNs). In particular, a siamese convolutional network is applied to obtain a weight map which integrates the pixel activity information from two source images. This CNN-based approach can deal with two vital issues in image fusion as a whole, namely, activity level measurement and weight assignment. Considering the different imaging modalities of infrared and visible images, the merging procedure is conducted in a multi-scale manner via image pyramids and a local similarity-based strategy is adopted to adaptively adjust the fusion mode for the decomposed coefficients. Experimental results demonstrate that the proposed method can achieve state-of-the-art results in terms of both visual quality and objective assessment.
Visible and Infrared Image Fusion Based on Attention and Multiscale Residuals

Zhongxu Xiang,Wentie Yang,Zuoshuai Wang,Yidong Xu,Vladimir Grischenko,Vladimir Korochentsev

DOI: https://doi.org/10.1117/12.3033768

2024-01-01

Abstract:Image fusion is a significant research area, related to a specific fusion task and has broad application prospects. Most of the existing image fusion algorithms are based on the pixel level, and although the convolutional network with more layers has a powerful feature extraction capability, its complexity increases with the deepening of the network. The dependency of the local range of the image is also not fully utilized in this process by the convolutional network, which makes the fused image detail information lost. In this paper, we propose a local-attention mechanism based network with multi-scale residuals to fuse visible and infrared images. The network consists of two key parts: encoder-decoder, fusion strategy. During the network training process, we utilize a phased training approach, where an automatic codec is first trained for conducting feature extraction, local feature enhancement and feature reconstruction. In the fusion stage, the coder trained in the first step is utilized to extract the two light image features, and then these features are fused by a multi-scale residual network, respectively. Finally, the fused features are inverse encoding to acquire the fused image. The experimental comparison results show that our fusion solution outperforms the existing fusion methods in both visual perception and objective evaluation.
A Semantic-Aware and Multi-Guided Network for Infrared-Visible Image Fusion

Xiaoli Zhang,Liying Wang,Libo Zhao,Xiongfei Li,Siwei Ma

2024-08-03

Abstract:Multi-modality image fusion aims at fusing specific-modality and shared-modality information from two source images. To tackle the problem of insufficient feature extraction and lack of semantic awareness for complex scenes, this paper focuses on how to model correlation-driven decomposing features and reason high-level graph representation by efficiently extracting complementary features and multi-guided feature aggregation. We propose a three-branch encoder-decoder architecture along with corresponding fusion layers as the fusion strategy. The transformer with Multi-Dconv Transposed Attention and Local-enhanced Feed Forward network is used to extract shallow features after the depthwise convolution. In the three parallel branches encoder, Cross Attention and Invertible Block (CAI) enables to extract local features and preserve high-frequency texture details. Base feature extraction module (BFE) with residual connections can capture long-range dependency and enhance shared-modality expression capabilities. Graph Reasoning Module (GR) is introduced to reason high-level cross-modality relations and extract low-level details features as CAI's specific-modality complementary information simultaneously. Experiments demonstrate that our method has obtained competitive results compared with state-of-the-art methods in visible/infrared image fusion and medical image fusion tasks. Moreover, we surpass other fusion methods in terms of subsequent tasks, averagely scoring 8.27% mAP@0.5 higher in object detection and 5.85% mIoU higher in semantic segmentation.

Computer Vision and Pattern Recognition
Effect of Laser Resurfacing on p53 Expression in Photoaged Facial Skin

M. El-Domyati,Sameh K. Attia,A. Esmat,Hesham M. Ahmad,Hossam M. ABDEL WAHAB,Belkais M Badr

DOI: https://doi.org/10.1111/J.1524-4725.2007.33141.X

2007-06-01

Dermatologic Surgery

Abstract:BACKGROUND p53 overexpression has been reported in photoaged skin. Meanwhile, p53 gene mutations have been implicated as an important factor in the pathogenesis of ultraviolet (UV) light–induced skin cancer. OBJECTIVE The objective was to evaluate the effect of laser resurfacing on the epidermal thickness and expression of p53 in photoaged skin. METHODS Specimens were obtained from the facial skin of 10 patients before and after 3 months and 1 year of treatment using CO2 (five cases) and erbium (Er):YAG (five cases) lasers. Specimens were also obtained from six age-matched controls. These biopsies were used for routine histopathology, histometry, and p53 immunoperoxidase staining. RESULTS Both CO2 and Er:YAG lasers were found to induce a significant decrease in p53 expression in biopsies obtained after 3 months (p=.0004 and .002, respectively) followed by gradual increase (p=.01 in both groups). A significant increase (p<.01) in epidermal thickness was also observed after 1 year of resurfacing. This increase, however, is inversely correlated with the level of p53 expression in such patients. CONCLUSION The decrease in epidermal p53 expression after CO2 and Er:YAG lasers may account for some of the benefits of resurfacing on the epidermis, as well as prevention of actinic neoplasia by adjusting any disturbance in the proliferation/apoptosis balance observed in photoaged facial skin.
MAFusion: Multiscale Attention Network for Infrared and Visible Image Fusion

Xiaoling Li,Houjin Chen,Yanfeng Li,Yahui Peng

DOI: https://doi.org/10.1109/tim.2022.3181898

IF: 5.6

2022-06-24

IEEE Transactions on Instrumentation and Measurement

Abstract:The infrared and visible image fusion aims to generate one image with rich information by integrating thermal regions from the infrared image and texture details from the visible image, which is beneficial to facilitate the capacity of video surveillance and object detection in complex environments. Although there is great progress in image fusion algorithms, artifacts and inconsistencies are still challenging tasks. To alleviate these problems, a multiscale attention network for infrared and visible image fusion (MAFusion) is proposed. The network consists of an encoder, a fusion strategy, and a decoder. Specifically, the encoder is adopted to extract multiscale features by feeding the source images. An attention-based model is then designed as the fusion strategy to integrate different features in the infrared and visible images. The attention-based model can highlight the thermal targets in the infrared image and maintain details in the visible image, so as to avoid the generation of artifacts. The decoder is based on a multiscale skip connection to incorporate low-level details with high-level semantics at different scales. The vital features of infrared and visible images can be fully preserved by the multiscale skip connection network to restrict the introduction of inconsistencies. Furthermore, we develop a feature-preserving loss function to train the proposed network. Experimental results demonstrate that the proposed network delivers advantages and effectiveness compared with the state-of-the-art fusion methods in qualitative and quantitative assessments. Besides, we apply the fused image generated by MAFusion to crowd counting (CC), which can effectively improve the CC performance in low-illumination conditions.

engineering, electrical & electronic,instruments & instrumentation
Recognition of Fusion Processing Based on Infrared and Visible Images under the Framework of Semi-supervised Learning

Zhuohang Cao,Shuhao Bian,Zhe Chen,Peili Ma,Shuaishuai Zhai,Lijun Xu

DOI: https://doi.org/10.1088/1742-6596/1550/3/032010

2020-05-01

Journal of Physics: Conference Series

Abstract:Abstract In traditional convolutional networks, due to the lack of adequate information protection of traditional image fusion technology and the incomplete removal of redundant noise, the useful information of the fusion image is missing and the recognition success rate is low. In this paper, through the research of deep learning-based image fusion methods and traditional target recognition and SVM neural network, an image fusion processing recognition method based on infrared and visible light is designed. The coding network of this image fusion method consists of convolutional layers, fusion layers and dense blocks. The output of each layer needs to be connected to the next few layers by using a densely connected neural network, so as to obtain more useful features from the source image and fuse the data of the two images better. It is verified by simulation that the fused image has sound visual effects, and its edges and details have been completely preserved. Thus, the target object has strong recognizability compared with the surrounding environment. Research shows that this method will help to more accurately interpret target information in complex environments and achieve more effective results in target recognition.
Interactive residual coordinate attention and contrastive learning for infrared and visible image fusion in triple frequency bands

Zhihua Xie,Sha Zong,Qiang Li,Peiqi Cai,Yaxiong Zhan,Guodong Liu

DOI: https://doi.org/10.1038/s41598-023-51045-9

IF: 4.6

2024-01-02

Scientific Reports

Abstract:Abstract The auto-encoder (AE) based image fusion models have achieved encouraging performance on infrared and visible image fusion. However, the meaningful information loss in the encoding stage and simple unlearnable fusion strategy are two significant challenges for such models. To address these issues, this paper proposes an infrared and visible image fusion model based on interactive residual attention fusion strategy and contrastive learning in the frequency domain. Firstly, the source image is transformed into three sub-bands of the high-frequency, low-frequency, and mid-frequency for powerful multiscale representation from the prospective of the frequency spectrum analysis. To further cope with the limitations of the straightforward fusion strategy, a learnable coordinate attention module in the fusion layer is incorporated to adaptively fuse representative information based on the characteristics of the corresponding feature maps. Moreover, the contrastive learning is leveraged to train the multiscale decomposition network for enhancing the complementarity of information at different frequency spectra. Finally, the detail-preserving loss, feature enhancing loss and contrastive loss are incorporated to jointly train the entire fusion model for good detail maintainability. Qualitative and quantitative comparisons demonstrate the feasibility and validity of our model, which can consistently generate fusion images containing both highlight targets and legible details, outperforming the state-of-the-art fusion methods.

multidisciplinary sciences
An end-to-end multi-scale network based on autoencoder for infrared and visible image fusion

Hongzhe Liu,Hua Yan

DOI: https://doi.org/10.1007/s11042-022-14314-9

IF: 2.577

2022-12-28

Multimedia Tools and Applications

Abstract:Infrared and visible image fusion aims to obtain a more informative fusion image by merging the infrared and visible images. However, the existing methods have some shortcomings, such as detail information loss, unclear boundaries, and not being end-to-end. In this paper, we propose an end-to-end network architecture for infrared and visible image fusion task. Our network contains three essential parts: encoders, residual fusion module, and decoder. First, we input infrared and visible images to two encoders to extract shallow features, respectively. Subsequently, the two sets of features are concatenated and fed to the residual fusion module to extract multi-scale features and fuse them adequately. Finally, the fused image is obtained by the decoder. We conduct objective and subjective experiments on two public datasets. The comparison results with the state-of-art methods prove that the fusion results of the proposed method have better objective metrics and contain more detail information and more explicit boundary.

computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering
SCFusion: Infrared and Visible Fusion Based on Salient Compensation

Haipeng Liu,Meiyan Ma,Meng Wang,Zhaoyu Chen,Yibo Zhao

DOI: https://doi.org/10.3390/e25070985

2023-06-27

Abstract:The aim of infrared and visible image fusion is to integrate the complementary information of the two modalities for high-quality fused images. However, many deep learning fusion algorithms have not considered the characteristics of infrared images in low-light scenes, leading to the problems of weak texture details, low contrast of infrared targets and poor visual perception in the existing methods. Therefore, in this paper, we propose a salient compensation-based fusion method that makes sufficient use of the characteristics of infrared and visible images to generate high-quality fused images under low-light conditions. First, we design a multi-scale edge gradient module (MEGB) in the texture mainstream to adequately extract the texture information of the dual input of infrared and visible images; on the other hand, the salient tributary is pre-trained by salient loss to obtain the saliency map based on the salient dense residual module (SRDB) to extract salient features, which is supplemented in the process of overall network training. We propose the spatial bias module (SBM) to fuse global information with local information. Finally, extensive comparison experiments with existing methods show that our method has significant advantages in describing target features and global scenes, the effectiveness of the proposed module is demonstrated by ablation experiments. In addition, we also verify the facilitation of this paper's method for high-level vision on a semantic segmentation task.
Correlation-Guided Discriminative Cross-Modality Features Network for Infrared and Visible Image Fusion

Zhao Cai,Yong Ma,Jun Huang,Xiaoguang Mei,Fan Fan

DOI: https://doi.org/10.1109/tim.2023.3341137

IF: 5.6

2024-01-01

IEEE Transactions on Instrumentation and Measurement

Abstract:Infrared and visible image fusion, which focuses on generating a new synthetic image, seeks to integrate the complementary properties of both modalities to contain abundant information. Many existing image fusion methods adopt artificially designed fusion rules. However, they often lack sufficient consideration for the interaction of long-range context information. As a result, important information may be lost, which limits the application of subsequent vision tasks. To address these limitations, we propose a novel unsupervised image fusion network named DCSFuse, which is guided by the correlation of image features. Our method can adaptively integrate complementary and long-range context information. More specifically, the proposed method first learns the modal-specific features of the two modalities. Then it calculates the correlation between each feature and integrates the cross-modal features guided by the correlation. Finally, these integrated features are reconstructed into a fused image that contains supplementary information from both modalities and captures long-range context dependencies. Extensive experiments on three mainstream datasets demonstrate the superiority of our method over other methods in terms of quantitative metrics and qualitative effects. Furthermore, we demonstrate experimentally that our fused images enhance the accuracy of visual object detection tasks. In particular, our network benefits from its superior computational efficiency, enabling it to generate fused images in real time. The source code of DCSFuse has been released at: https://github.com/zc617/DCSFuse.

engineering, electrical & electronic,instruments & instrumentation
SimpleFusion: A Simple Fusion Framework for Infrared and Visible Images

Ming Chen,Yuxuan Cheng,Xinwei He,Xinyue Wang,Yan Aze,Jinhai Xiang

2024-06-27

Abstract:Integrating visible and infrared images into one high-quality image, also known as visible and infrared image fusion, is a challenging yet critical task for many downstream vision tasks. Most existing works utilize pretrained deep neural networks or design sophisticated frameworks with strong priors for this task, which may be unsuitable or lack flexibility. This paper presents SimpleFusion, a simple yet effective framework for visible and infrared image fusion. Our framework follows the decompose-and-fusion paradigm, where the visible and the infrared images are decomposed into reflectance and illumination components via Retinex theory and followed by the fusion of these corresponding elements. The whole framework is designed with two plain convolutional neural networks without downsampling, which can perform image decomposition and fusion efficiently. Moreover, we introduce decomposition loss and a detail-to-semantic loss to preserve the complementary information between the two modalities for fusion. We conduct extensive experiments on the challenging benchmarks, verifying the superiority of our method over previous state-of-the-arts. Code is available at \href{<a class="link-external link-https" href="https://github.com/hxwxss/SimpleFusion-A-Simple-Fusion-Framework-for-Infrared-and-Visible-Images" rel="external noopener nofollow">this https URL</a>}{<a class="link-external link-https" href="https://github.com/hxwxss/SimpleFusion-A-Simple-Fusion-Framework-for-Infrared-and-Visible-Images" rel="external noopener nofollow">this https URL</a>}

Computer Vision and Pattern Recognition
Multi-scale attention-based lightweight network with dilated convolutions for infrared and visible image fusion

Fuquan Li,Yonghui Zhou,YanLi Chen,Jie Li,ZhiCheng Dong,Mian Tan

DOI: https://doi.org/10.1007/s40747-023-01185-2

IF: 6.7

2023-08-03

Complex & Intelligent Systems

Abstract:Abstract Infrared and visible image fusion aims to generate synthetic images including salient targets and abundant texture details. However, traditional techniques and recent deep learning-based approaches have faced challenges in preserving prominent structures and fine-grained features. In this study, we propose a lightweight infrared and visible image fusion network utilizing multi-scale attention modules and hybrid dilated convolutional blocks to preserve significant structural features and fine-grained textural details. First, we design a hybrid dilated convolutional block with different dilation rates that enable the extraction of prominent structure features by enlarging the receptive field in the fusion network. Compared with other deep learning methods, our method can obtain more high-level semantic information without piling up a large number of convolutional blocks, effectively improving the ability of feature representation. Second, distinct attention modules are designed to integrate into different layers of the network to fully exploit contextual information of the source images, and we leverage the total loss to guide the fusion process to focus on vital regions and compensate for missing information. Extensive qualitative and quantitative experiments demonstrate the superiority of our proposed method over state-of-the-art methods in both visual effects and evaluation metrics. The experimental results on public datasets show that our method can improve the entropy (EN) by 4.80%, standard deviation (SD) by 3.97%, correlation coefficient (CC) by 1.86%, correlations of differences (SCD) by 9.98%, and multi-scale structural similarity (MS_SSIM) by 5.64%, respectively. In addition, experiments with the VIFB dataset further indicate that our approach outperforms other comparable models.

computer science, artificial intelligence
Advancing infrared and visible image fusion with an enhanced multiscale encoder and attention-based networks

Jiashuo Wang,Yong Chen,Xiaoyun Sun,Hui Xing,Fan Zhang,Shiji Song,Shuyong Yu

DOI: https://doi.org/10.1016/j.isci.2024.110915

IF: 5.8

2024-09-10

iScience

Abstract:Infrared and visible image fusion aims to produce images that highlight key targets and offer distinct textures, by merging the thermal radiation infrared images with the detailed texture visible images. Traditional auto encoder-decoder-based fusion methods often rely on manually designed fusion strategies, which lack flexibility across different scenarios. Addressing this limitation, we introduce EMAFusion, a fusion approach featuring an enhanced multiscale encoder and a learnable, lightweight fusion network. Our method incorporates skip connections, the convolutional block attention module (CBAM), and nest architecture within the auto encoder-decoder framework to adeptly extract and preserve multiscale features for fusion tasks. Furthermore, a fusion network driven by spatial and channel attention mechanisms is proposed, designed to precisely capture and integrate essential features from both image types. Comprehensive evaluations of the TNO image fusion dataset affirm the proposed method's superiority over existing state-of-the-art techniques, demonstrating its potential for advancing infrared and visible image fusion.
Multi-scale unsupervised network for infrared and visible image fusion based on joint attention mechanism

Dongdong Xu,Ning Zhang,Yuxi Zhang,Zheng Li,Zhikang Zhao,Yongcheng Wang

DOI: https://doi.org/10.1016/j.infrared.2022.104242

2022-09-01

Abstract:Infrared and visible image fusion can synthesize complementary features of salient objects and texture details which are important for all-weather detection and other tasks. Nowadays, the deep learning based unsupervised fusion solutions are preferred and have obtained good results since the reference images for fusion tasks are not available. In the existing methods, some prominent features are missing in the fused images and the visual vitality needs to be improved. From this thought, attention mechanism is introduced to the fusion network. Especially, channel dimension and spatial dimension attention are jointed to supplement each other for feature extraction. Multiple attention branches emphasize on multi-scale features to complete the encoding. Skip connections are added to learn residual information. The multi-layer perceptual loss, the structure similarity loss and the content loss together construct the strong constraints for training. Comparative experiments with subjective and objective evaluations on 4 traditional and 9 deep learning based methods demonstrate the advantages of the proposed model.

optics,physics, applied,instruments & instrumentation

The Milky Way: From Mammary Gland to Milk to Newborn

Fusion of Low-Illuminance Visible and Near-Infrared Images Based on Convolutional Neural Networks

Infrared and Visible Image Fusion Based on a Two-Stage Class Conditioned Auto-Encoder Network.

Fusion of Infrared and Visible Images Via Multi-Layer Convolutional Sparse Representation

Infrared and visible image fusion with entropy-based adaptive fusion module and mask-guided convolutional neural network

EV-Fusion: A Novel Infrared and Low-Light Color Visible Image Fusion Network Integrating Unsupervised Visible Image Enhancement

A Multi-Stage Visible and Infrared Image Fusion Network Based on Attention Mechanism

Infrared and Visible Image Fusion with Convolutional Neural Networks.

Visible and Infrared Image Fusion Based on Attention and Multiscale Residuals

A Semantic-Aware and Multi-Guided Network for Infrared-Visible Image Fusion

Effect of Laser Resurfacing on p53 Expression in Photoaged Facial Skin

MAFusion: Multiscale Attention Network for Infrared and Visible Image Fusion

Recognition of Fusion Processing Based on Infrared and Visible Images under the Framework of Semi-supervised Learning

Interactive residual coordinate attention and contrastive learning for infrared and visible image fusion in triple frequency bands

An end-to-end multi-scale network based on autoencoder for infrared and visible image fusion

SCFusion: Infrared and Visible Fusion Based on Salient Compensation

Correlation-Guided Discriminative Cross-Modality Features Network for Infrared and Visible Image Fusion

SimpleFusion: A Simple Fusion Framework for Infrared and Visible Images

Multi-scale attention-based lightweight network with dilated convolutions for infrared and visible image fusion

Advancing infrared and visible image fusion with an enhanced multiscale encoder and attention-based networks

Multi-scale unsupervised network for infrared and visible image fusion based on joint attention mechanism