Transplant-related Kaposi sarcoma in children.

M. Al-Sulaiman,D. Mousa,Z. Rassoul,A. Abdalla,M. Abdur Rehman,A. Al-Khader

DOI: https://doi.org/10.1093/NDT/9.4.443

Abstract:

What problem does this paper attempt to address?

Multimodal medical image fusion and classification using deep learning techniques

D. Veeraiah,S. Sai Kumar,Rajendra Kumar Ganiya,Katta Subba Rao,J. Nageswara Rao,Ramaswamy Manjith,A. Rajaram

DOI: https://doi.org/10.3233/jifs-240018

2024-04-20

Journal of Intelligent & Fuzzy Systems

Abstract:Medical image fusion plays a crucial role in accurate medical diagnostics by combining images from various modalities. To address this need, we propose an AI model for efficient medical image fusion using multiple modalities. Our approach utilizes a Siamese convolutional neural network to construct a weight map based on pixel movement information extracted from multimodality medical images. We leverage medical picture pyramids to incorporate multiscale techniques, enhancing reliability beyond human visual intuition. Additionally, we dynamically adjust the fusion mode based on local comparisons of deconstructed coefficients. Evaluation metrics including F1-score, recall, accuracy, and precision are computed to assess performance, yielding impressive results: an F1-score of 0.8551 and a mutual information (MI) value of 2.8059. Experimental results demonstrate the superiority of our method, achieving a remarkable 99.61% accuracy in targeted experiments. Moreover, the Structural Similarity Index (SSIM) of our approach is 0.8551. Compared to state-of-the-art approaches, our model excels in medical picture classification, providing accurate diagnosis through high-quality fused images. This research advances medical image fusion techniques, offering a robust solution for precise medical diagnostics across various modalities.
An efficient Dense-Resnet for multimodal image fusion using medical image

Tanima Ghosh,N. Jayanthi,Ghosh, Tanima

DOI: https://doi.org/10.1007/s11042-024-18974-7

IF: 2.577

2024-04-02

Multimedia Tools and Applications

Abstract:Nowadays, the visual content of various medical images is increased through multimodal image fusion to gather more information from medical images. The complementary information available in various modalities is merged to increase the visual content of the image for quick medical diagnosis. However, the resultant fused multi-modality images suffered from different issues, like texture distortion and gradient, mainly for the affected region. Thus, a hybrid deep learning model, Dense-ResNet is designed for the fusing of multimodal medical images from different modalities in this research work. The images from three different modalities are collected initially, which are individually pre-processed by using a median filter. The pre-processed image of the spatial domain is transformed into a spectral domain by applying Dual-Tree Complex Wavelet Transform (DTCWT), and the transformed image is segmented using Edge-Attention Guidance Network (ET-Net). Finally, the multimodal fusion of medical images is performed on the segmented images using the designed Dense-ResNet model. Moreover, the superiority of the designed model is validated, which shows that the designed Dense-ResNet model outperforms as compared with other existing multimodal medical image fusion approaches. The Dense-ResNet model achieved 0.402 Mean Square Error (MSE), 0.634 Root Mean Square Error (RMSE), and 47.136 dB Peak Signal to Noise Ratio (PSNR).

computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering
Adaptive Multimodal Image Fusion with a Deep Pyramidal Residual Learning Network

Kiranmai Bellam,N. Krishnaraj,T. Jayasankar,N. B. Prakash,G. R. Hemalakshmi

DOI: https://doi.org/10.1166/jmihi.2021.3763

2021-08-01

Journal of Medical Imaging and Health Informatics

Abstract:Multimodal medical imaging is an indispensable requirement in the treatment of various pathologies to accelerate care. Rather than discrete images, a composite image combining complementary features from multimodal images is highly informative for clinical examinations, surgical planning, and progress monitoring. In this paper, a deep learning fusion model is proposed for the fusion of medical multimodal images. Based on pyramidal and residual learning units, the proposed model, strengthened with adaptive fusion rules, is tested on image pairs from a standard dataset. The potential of the proposed model for enhanced image exams is shown by fusion studies with deep network images and quantitative output metrics of magnetic resonance imaging and positron emission tomography (MRI/PET) and magnetic resonance imaging and single-photon emission computed tomography (MRI/SPECT). The proposed fusion model achieves the Structural Similarity Index Measure (SSIM) values of 0.9502 and 0.8103 for the MRI/SPECT and MRI/PET MRI/SPECT image sets, signifying the perceptual visual consistency of the fused images. Testing is performed on 20 pairs of MRI/SPECT and MRI/PET images. Similarly, the Mutual Information (MI) values of 2.7455 and 2.7776 obtained for the MRI/SPECT and MRI/PET image sets, indicating the model’s ability to capture the information content from the source images to the composite image. Further, the proposed model allows deploying its variants, introducing refinements on the basic model suitable for the fusion of low and high-resolution medical images of diverse modalities.
A Medical Image Fusion Method Based on Convolutional Neural Networks

Yu Liu,Xun Chen,Juan Cheng,Hu Peng

DOI: https://doi.org/10.23919/icif.2017.8009769

2017-01-01

Abstract:Medical image fusion technique plays an an increasingly critical role in many clinical applications by deriving the complementary information from medical images with different modalities. In this paper, a medical image fusion method based on convolutional neural networks (CNNs) is proposed. In our method, a siamese convolutional network is adopted to generate a weight map which integrates the pixel activity information from two source images. The fusion process is conducted in a multi-scale manner via image pyramids to be more consistent with human visual perception. In addition, a local similarity based strategy is applied to adaptively adjust the fusion mode for the decomposed coefficients. Experimental results demonstrate that the proposed method can achieve promising results in terms of both visual quality and objective assessment.
Multi-modal Medical Neurological Image Fusion using Wavelet Pooled Edge Preserving Autoencoder

Manisha Das,Deep Gupta,Petia Radeva,Ashwini M Bakde

2023-10-18

Abstract:Medical image fusion integrates the complementary diagnostic information of the source image modalities for improved visualization and analysis of underlying anomalies. Recently, deep learning-based models have excelled the conventional fusion methods by executing feature extraction, feature selection, and feature fusion tasks, simultaneously. However, most of the existing convolutional neural network (CNN) architectures use conventional pooling or strided convolutional strategies to downsample the feature maps. It causes the blurring or loss of important diagnostic information and edge details available in the source images and dilutes the efficacy of the feature extraction process. Therefore, this paper presents an end-to-end unsupervised fusion model for multimodal medical images based on an edge-preserving dense autoencoder network. In the proposed model, feature extraction is improved by using wavelet decomposition-based attention pooling of feature maps. This helps in preserving the fine edge detail information present in both the source images and enhances the visual perception of fused images. Further, the proposed model is trained on a variety of medical image pairs which helps in capturing the intensity distributions of the source images and preserves the diagnostic information effectively. Substantial experiments are conducted which demonstrate that the proposed method provides improved visual and quantitative results as compared to the other state-of-the-art fusion methods.

Image and Video Processing,Computer Vision and Pattern Recognition,Machine Learning
Multimodal Medical Image Fusion: The Perspective of Deep Learning

Mingyang Wei,Mengbo Xi,Yabei Li,Minjun Liang,Ge Wang

DOI: https://doi.org/10.54097/ajst.v5i3.8013

2023-05-05

Academic Journal of Science and Technology

Abstract:Multimodal medical image fusion involves the integration of medical images originating from distinct modalities and captured by various sensors, with the aim to enhance image quality, minimize redundant information, and preserve specific features, ultimately leading to increased efficiency and accuracy in clinical diagnoses. In recent years, the emergence of deep learning techniques has propelled significant advancements in image fusion, addressing the limitations of conventional methods that necessitate manual design of activity level measurement and fusion rules. This paper initially presents a systematic description of the multimodal medical image fusion problem, delineating the interrelationships between different fusion modalities while summarizing their characteristics and functions. Subsequently, it reviews the theories and enhancement approaches associated with deep learning in the medical image fusion domain, striving for a comprehensive overview of the state-of-the-art developments in this field from a deep learning perspective. These developments encompass multimodal feature extraction methods based on convolutional techniques, adversarial learning-based methods, convolutional sparse representation and stacked autoencoder-based signal processing methods, and unified models. Lastly, the paper summarizes the enhancement techniques for multimodal medical image fusion methods, highlighting the pressing issues and challenges encountered by deep learning approaches in this domain.
Medical image fusion with deep neural networks

Nannan Liang

DOI: https://doi.org/10.1038/s41598-024-58665-9

IF: 4.6

2024-04-05

Scientific Reports

Abstract:Medical image fusion aims to fuse multiple images from a single or multiple imaging modes to enhance their corresponding clinical applications in diagnosing and evaluating medical problems, a trend that has attracted increasing attention. However, most recent medical image fusion methods require prior knowledge, making it difficult to select image features. In this paper, we propose a novel deep medical image fusion method based on a deep convolutional neural network (DCNN) for directly learning image features from original images. Specifically, source images are first decomposed by low rank representation to obtain the principal and salient components, respectively. Following that, the deep features are extracted from the decomposed principal components via DCNN and fused by a weighted-average rule. Then, considering the complementary between the salient components obtained by the low rank representation, a simple yet effective sum rule is designed to fuse the salient components. Finally, the fused result is obtained by reconstructing the principal and salient components. The experimental results demonstrate that the proposed method outperforms several state-of-the-art medical image fusion approaches in terms of both objective indices and visual quality.

multidisciplinary sciences
Feature extraction of multimodal medical image fusion using novel deep learning and contrast enhancement method

Jameel Ahmed Bhutto,Jiang Guosong,Ziaur Rahman,Muhammad Ishfaq,Zhengzheng Sun,Toufique Ahmed Soomro

DOI: https://doi.org/10.1007/s10489-024-05431-z

IF: 5.3

2024-05-02

Applied Intelligence

Abstract:The fusion of multimodal medical images has garnered painstaking attention for clinical diagnosis and surgical planning. Although various scholars have designed numerous fusion methods, the challenges of extracting substantial features without introducing noise and non-uniform contrast hindered the overall quality of fused photos. This paper presents a multimodal medical image fusion (MMIF) using a novel deep convolutional neural network (D-CNN) along with preprocessing schemes to circumvent the mentioned issues. A non-linear average median filtering (NL-AMF) and multiscale improved top-hat (MI-TH) approach are utilized at the preprocessing stage to remove noise and improve the contrast of images. The non-linear anisotropic diffusion (NL-AD) scheme is employed to split the photos into base and detailed parts. The fusion of base parts is accomplished by a dimension reduction method to retain the energy information. In contrast, the detailed parts are fused by novel D-CNN to preserve the enriched detailed features effectively. The simulation results demonstrate that the proposed method produces better brightness contrast and more image details than existing methods by acquiring 0.7649 to 0.8986 , 0.3520 to 0.4783, 0.7639 to 0.9056, 68.8932 to 81.0487 gain for quality transfer ratio from source photo to a generated photo ( ), feature mutual information (FMI), structural similarity index (SSIM), and average pixel intensity (API) respectively.

computer science, artificial intelligence
Multimodal medical image fusion based on interval gradients and convolutional neural networks

Xiaolong Gu,Ying Xia,Jie Zhang

DOI: https://doi.org/10.1186/s12880-024-01418-x

2024-09-05

Abstract:Many image fusion methods have been proposed to leverage the advantages of functional and anatomical images while compensating for their shortcomings. These methods integrate functional and anatomical images while presenting physiological and metabolic organ information, making their diagnostic efficiency far greater than that of single-modal images. Currently, most existing multimodal medical imaging fusion methods are based on multiscale transformation, which involves obtaining pyramid features through multiscale transformation. Low-resolution images are used to analyse approximate image features, and high-resolution images are used to analyse detailed image features. Different fusion rules are applied to achieve feature fusion at different scales. Although these fusion methods based on multiscale transformation can effectively achieve multimodal medical image fusion, much detailed information is lost during multiscale and inverse transformation, resulting in blurred edges and a loss of detail in the fusion images. A multimodal medical image fusion method based on interval gradients and convolutional neural networks is proposed to overcome this problem. First, this method uses interval gradients for image decomposition to obtain structure and texture images. Second, deep neural networks are used to extract perception images. Three methods are used to fuse structure, texture, and perception images. Last, the images are combined to obtain the final fusion image after colour transformation. Compared with the reference algorithms, the proposed method performs better in multiple objective indicators of Q EN , Q NIQE , Q SD , Q SSEQ and Q TMQI .
Multimodal medical image fusion using convolutional neural network and extreme learning machine

Weiwei Kong,Chi Li,Yang Lei

DOI: https://doi.org/10.3389/fnbot.2022.1050981

IF: 3.493

2022-11-17

Frontiers in Neurorobotics

Abstract:The emergence of multimodal medical imaging technology greatly increases the accuracy of clinical diagnosis and etiological analysis. Nevertheless, each medical imaging modal unavoidably has its own limitations, so the fusion of multimodal medical images may become an effective solution. In this paper, a novel fusion method on the multimodal medical images exploiting convolutional neural network (CNN) and extreme learning machine (ELM) is proposed. As a typical representative in deep learning, CNN has been gaining more and more popularity in the field of image processing. However, CNN often suffers from several drawbacks, such as high computational costs and intensive human interventions. To this end, the model of convolutional extreme learning machine (CELM) is constructed by incorporating ELM into the traditional CNN model. CELM serves as an important tool to extract and capture the features of the source images from a variety of different angles. The final fused image can be obtained by integrating the significant features together. Experimental results indicate that, the proposed method is not only helpful to enhance the accuracy of the lesion detection and localization, but also superior to the current state-of-the-art ones in terms of both subjective visual performance and objective criteria.

robotics,computer science, artificial intelligence,neurosciences
Multimodal Medical Image Fusion Using Stacked Auto-encoder in NSCT Domain

Nahed Tawfik,Heba A. Elnemr,Mahmoud Fakhr,Moawad I. Dessouky,Fathi E. Abd El-Samie

DOI: https://doi.org/10.1007/s10278-021-00554-y

IF: 4.903

2022-07-01

Journal of Digital Imaging

Abstract:Medical image fusion is a process that aims to merge the important information from images with different modalities of the same organ of the human body to create a more informative fused image. In recent years, deep learning (DL) methods have achieved significant breakthroughs in the field of image fusion because of their great efficiency. The DL methods in image fusion have become an active topic due to their high feature extraction and data representation ability. In this work, stacked sparse auto-encoder (SSAE), a general category of deep neural networks, is exploited in medical image fusion. The SSAE is an efficient technique for unsupervised feature extraction. It has high capability of complex data representation. The proposed fusion method is carried as follows. Firstly, the source images are decomposed into low- and high-frequency coefficient sub-bands with the non-subsampled contourlet transform (NSCT). The NSCT is a flexible multi-scale decomposition technique, and it is superior to traditional decomposition techniques in several aspects. After that, the SSAE is implemented for feature extraction to obtain a sparse and deep representation from high-frequency coefficients. Then, the spatial frequencies are computed for the obtained features to be used for high-frequency coefficient fusion. After that, a maximum-based fusion rule is applied to fuse the low-frequency sub-band coefficients. The final integrated image is acquired by applying the inverse NSCT. The proposed method has been applied and assessed on various groups of medical image modalities. Experimental results prove that the proposed method could effectively merge the multimodal medical images, while preserving the detail information, perfectly.

radiology, nuclear medicine & medical imaging
A novel framework of multimodal medical image fusion using adaptive NSST and optimized deep learning approach

K. Vanitha,D. Satyanarayana,M. N. GiriPrasad

DOI: https://doi.org/10.1080/13682199.2023.2241793

2023-08-16

The Imaging Science Journal

Abstract:Multimodal medical image fusion plays a pivotal role in the medical and imaging industry. Existing works of deep learning method suffers from blurred texture characteristics and computing efficiency. Thus, a novel deep learning model is proposed for multimodal medical image fusion. Initially, an Adaptive Non-Subsampled Shearlet Transform (ANSST) approach is developed for decomposing the images, where the filter coefficient is optimized by Hybrid Water Strider-Dingo Optimization (HWS-DOX). The fusion of the high sub-bands of source image 1 and high sub-bands of source image 2 is done by an Optimized Deep Neural Network (ODNN). Then, the lower sub-bands of source image 1 and lower sub-bands of source image 2 will be fused through a weighted averaging scheme. Finally, the final fused images are attained by applying the inverse ANSST. Thus, the performance of recommended module is compared over classical heuristics and various transform methods.

imaging science & photographic technology
Multi View Image Fusion Using Ensemble Deep Learning Algorithm for Mri and CT Images

N. Thenmoezhi,B. Perumal,A. Lakshmi

DOI: https://doi.org/10.1145/3640811

IF: 1.471

2024-01-31

ACM Transactions on Asian and Low-Resource Language Information Processing

Abstract:Medical image fusions are crucial elements in image based health care diagnostics or therapies, and generically applications of computer visions. However, majority of existing methods suffer from noise distortion that affect the overall output. When pictures are distorted by noises, classical fusion techniques perform badly. Hence, fusion techniques that properly maintain information comprehensively from multiple faulty pictures need to be created. This work presents ESLOs (Enhanced Lion Swarm Optimizations) with EDL (Ensemble Deep Learning) to address the aforementioned issues. The primary steps in this study include image fusions, segmentation, noise reduction, feature extraction, picture classification, and feature selection.AMFs (Adaptive Median Filters) are first used for noise removal in sequence to enhance image quality by eliminating noises. The MRIs and CTS images are then segmented using the RKMC algorithm to separate the images into their component regions or objects. Images in black and white are divided into image. In the white image, the RKMC algorithm successfully considered the earlier tumour probability. The next step is feature extraction, which is accomplished by using the MPCA (Modified Principal Component Analysis) to draw out the most informative aspects of the images. Then, ELSOs algorithm is applied for optimal feature selection which is computed by best fitness values. After that, multi view image fusions of multi modal images derive lower, middle and higher level images contents. It is done by using DCNNs (Deep Convolution Neural Networks) and TAcGANs (Tissue-Aware conditional Generative Adversarial Networks) algorithm which fuses the multi view features and relevant image features and it is used for real time applications. The results of this study implies that proposed ELSO+EDL algorithm results in better performances in terms of higher values of accuracies, PSNR and lower RMSE, MAPE with faster executions when compared to other existing algorithms.

computer science, artificial intelligence
Adaptive spatial and frequency experts fusion network for medical image fusion

Xianming Gu,Lihui Wang,Zeyu Deng,Ying Cao,Xingyu Huang,Yue-min Zhu

DOI: https://doi.org/10.1016/j.bspc.2024.106478

IF: 5.1

2024-06-02

Biomedical Signal Processing and Control

Abstract:Multi-modal medical image fusion is essential for the precise clinical diagnosis and surgical navigation since it can merge the complementary information in multi-modalities into a single image. Although existing deep learning-based fusion methods can fully exploit the semantic features of each modality, they cannot fuse them adaptively according to the importance of local and global features of each modality. To address this issue, we propose an adaptive spatial and frequency experts fusion network (ASFE-Fusion) for medical image fusion. Specifically, a cross attention spatial fusion (CASF) module is devised to adaptively fuse local features, which can constraint the consistency in spatial correlations by crossing the self-similarity map between two source images and therefore improve the details of the fused images. To complement the global information that may be lost in the CASF module, adaptive frequency fusion (AFF) module is introduced to fuse the features from a global perspective in frequency domain, which is useful for dealing with issue of intensity distortion. Subsequently, taking the fused features in spatial and frequency domains as two experts, an ensemble-learning-based spatial and frequency experts fusion (SFEF) module is used to adaptively fuse the local and global features. Finally, the fused image can be obtained from the fused features through a decoder by minimizing the content, gradient, and structure loss simultaneously. Extensive comparison experiments demonstrate that the proposed method outperforms state-of-the-art methods in terms of both visual quality and quantitative assessment. In addition, the downstream disease classification task also demonstrates the superiority of the proposed method.

engineering, biomedical
Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks

Yu Liu,Yu Shi,Fuhao Mu,Juan Cheng,Chang Li,Xun Chen

DOI: https://doi.org/10.1109/tim.2022.3184360

IF: 5.6

2022-07-08

IEEE Transactions on Instrumentation and Measurement

Abstract:Medical image fusion aims to integrate the complementary information captured by images of different modalities into a more informative composite image. However, current study on medical image fusion suffers from several drawbacks: 1) existing methods are mostly designed for 2-D slice fusion, and they tend to lose spatial contextual information when fusing medical images with volumetric structure slice by slice individually; 2) the few existing 3-D medical image fusion methods fail in considering the characteristics of source modalities sufficiently, leading to the loss of important modality information; and 3) most existing works concentrate on pursuing good performance on visual perception and objective evaluation, while there is a severe lack of clinical problem-oriented study. In this article, to address these issues, we propose a multimodal MRI volumetric data fusion method based on an end-to-end convolutional neural network (CNN). In our network, an attention-based multimodal feature fusion (MMFF) module is presented for more effective feature learning. In addition, a specific loss function that considers the characteristics of different MRI modalities is designed to preserve the modality information. Experimental results demonstrate that the proposed method can obtain more competitive results on both visual quality and objective assessment when compared with some representative 3-D and 2-D medical image fusion methods. We further verify the significance of the proposed method for brain tumor segmentation by enriching the input modalities, and the results show that it is helpful to improve the segmentation accuracy. The source code of our fusion method is available at https://github.com/yuliu316316/3D-CNN-Fusion.

engineering, electrical & electronic,instruments & instrumentation
Multimodal medical image fusion using residual network 50 in non subsampled contourlet transform

K. Koteswara Rao,K. Veera Swamy

DOI: https://doi.org/10.1080/13682199.2023.2175426

2023-06-21

The Imaging Science Journal

Abstract:Medical image fusion technology and its collective diagnosis are becoming crucial day by day. This task confers the latest algorithm for image fusion of medical images to many diagnostic complications. Firstly, transform is employed on input source images. The result of the application of transform is the decomposition of source images into various subbands. Eminent features are extracted from these subbands by using resnet50. These features are fused by phase congruency and guided filtering fusion rules. Finally, inverse transform gives the original image. The experiment results of this algorithm are compared with different methods by taking some pairs of medical images. Subjective and objective outcomes prove that the proposed algorithm exceeds the current methods by giving optimal performance measures in the area of medical diagnosis. Thus, it is revealed that the suggested multimodal image fusion model provides elevated performance over existing models via diverse diseases using MRI-SPECT and MRI-PET.

imaging science & photographic technology
Multi-modal medical image fusion using improved dual-channel PCNN

Adarsh Sinha,Rahul Agarwal,Vinay Kumar,Nitin Garg,Dhruv Singh Pundir,Harsimran Singh,Ritu Rani,Chinmaya Panigrahy

DOI: https://doi.org/10.1007/s11517-024-03089-w

Abstract:This paper proposes a medical image fusion method in the non-subsampled shearlet transform (NSST) domain to combine a gray-scale image with the respective pseudo-color image obtained through different imaging modalities. The proposed method applies a novel improved dual-channel pulse-coupled neural network (IDPCNN) model to fuse the high-pass sub-images, whereas the Prewitt operator is combined with maximum regional energy (MRE) to construct the fused low-pass sub-image. First, the gray-scale image and luminance of the pseudo-color image are decomposed using NSST to find the respective sub-images. Second, the low-pass sub-images are fused by the Prewitt operator and MRE-based rule. Third, the proposed IDPCNN is utilized to get the fused high-pass sub-images from the respective high-pass sub-images. Fourth, the luminance of the fused image is obtained by applying inverse NSST on the fused sub-images, which is combined with the chrominance components of the pseudo-color image to construct the fused image. A total of 28 diverse medical image pairs, 11 existing methods, and nine objective metrics are used in the experiment. Qualitative and quantitative fusion results show that the proposed method is competitive with and even outpaces some of the existing medical fusion approaches. It is also shown that the proposed method efficiently combines two gray-scale images.
Edge-Enhanced Dilated Residual Attention Network for Multimodal Medical Image Fusion

Meng Zhou,Yuxuan Zhang,Xiaolan Xu,Jiayi Wang,Farzad Khalvati

2024-11-19

Abstract:Multimodal medical image fusion is a crucial task that combines complementary information from different imaging modalities into a unified representation, thereby enhancing diagnostic accuracy and treatment planning. While deep learning methods, particularly Convolutional Neural Networks (CNNs) and Transformers, have significantly advanced fusion performance, some of the existing CNN-based methods fall short in capturing fine-grained multiscale and edge features, leading to suboptimal feature integration. Transformer-based models, on the other hand, are computationally intensive in both the training and fusion stages, making them impractical for real-time clinical use. Moreover, the clinical application of fused images remains unexplored. In this paper, we propose a novel CNN-based architecture that addresses these limitations by introducing a Dilated Residual Attention Network Module for effective multiscale feature extraction, coupled with a gradient operator to enhance edge detail learning. To ensure fast and efficient fusion, we present a parameter-free fusion strategy based on the weighted nuclear norm of softmax, which requires no additional computations during training or inference. Extensive experiments, including a downstream brain tumor classification task, demonstrate that our approach outperforms various baseline methods in terms of visual quality, texture preservation, and fusion speed, making it a possible practical solution for real-world clinical applications. The code will be released at <a class="link-external link-https" href="https://github.com/simonZhou86/en_dran" rel="external noopener nofollow">this https URL</a>.

Image and Video Processing,Artificial Intelligence,Computer Vision and Pattern Recognition
Multi-modal medical image fusion based on densely-connected high-resolution CNN and hybrid transformer

Quan Zhou,Shaozhuang Ye,Mingwei Wen,Zhiwen Huang,Mingyue Ding,Xuming Zhang

DOI: https://doi.org/10.1007/s00521-022-07635-1

2022-07-29

Neural Computing and Applications

Abstract:Multi-modal medical image fusion (MMIF) has found wide application in the field of disease diagnosis and surgical guidance. Despite the popularity of deep learning (DL)-based fusion methods, these DL algorithms cannot provide satisfactory fusion performance due to the difficulty in capturing the local information and the long-range dependencies effectively. To address these issues, this paper has presented an unsupervised MMIF method by combining a densely-connected high-resolution network (DHRNet) with a hybrid transformer. In this method, the local features are firstly extracted from the source image using the DHRNet. Then these features are input into the fine-grained attention module in the hybrid transformer to produce the global features by exploring their long-range dependencies. The local and global features are fused by the projection attention module in the hybrid transformer. Finally, based on the fused features, the fused result is reconstructed by the decoder network. The presented network is trained using an unsupervised loss function including edge preservation value, structural similarity, sum of the correlations of differences and structural tensor. Experiments on various multi-modal medical images show that, compared with several traditional and DL-based fusion methods, the presented method can generate visually better fused results and provide better quantitative metrics values.

computer science, artificial intelligence
Medical Image Segmentation Based on Multi-Modal Convolutional Neural Network: Study on Image Fusion Schemes

Zhe Guo,Xiang Li,Heng Huang,Ning Guo,Quanzheng Li

DOI: https://doi.org/10.1109/ISBI.2018.8363717

2017-11-03

Abstract:Image analysis using more than one modality (i.e. multi-modal) has been increasingly applied in the field of biomedical imaging. One of the challenges in performing the multimodal analysis is that there exist multiple schemes for fusing the information from different modalities, where such schemes are application-dependent and lack a unified framework to guide their designs. In this work we firstly propose a conceptual architecture for the image fusion schemes in supervised biomedical image analysis: fusing at the feature level, fusing at the classifier level, and fusing at the decision-making level. Further, motivated by the recent success in applying deep learning for natural image analysis, we implement the three image fusion schemes above based on the Convolutional Neural Network (CNN) with varied structures, and combined into a single framework. The proposed image segmentation framework is capable of analyzing the multi-modality images using different fusing schemes simultaneously. The framework is applied to detect the presence of soft tissue sarcoma from the combination of Magnetic Resonance Imaging (MRI), Computed Tomography (CT) and Positron Emission Tomography (PET) images. It is found from the results that while all the fusion schemes outperform the single-modality schemes, fusing at the feature level can generally achieve the best performance in terms of both accuracy and computational cost, but also suffers from the decreased robustness in the presence of large errors in any image modalities.

Computer Vision and Pattern Recognition,Machine Learning

Transplant-related Kaposi sarcoma in children.

Multimodal medical image fusion and classification using deep learning techniques

An efficient Dense-Resnet for multimodal image fusion using medical image

Adaptive Multimodal Image Fusion with a Deep Pyramidal Residual Learning Network

A Medical Image Fusion Method Based on Convolutional Neural Networks

Multi-modal Medical Neurological Image Fusion using Wavelet Pooled Edge Preserving Autoencoder

Multimodal Medical Image Fusion: The Perspective of Deep Learning

Medical image fusion with deep neural networks

Feature extraction of multimodal medical image fusion using novel deep learning and contrast enhancement method

Multimodal medical image fusion based on interval gradients and convolutional neural networks

Multimodal medical image fusion using convolutional neural network and extreme learning machine

Multimodal Medical Image Fusion Using Stacked Auto-encoder in NSCT Domain

A novel framework of multimodal medical image fusion using adaptive NSST and optimized deep learning approach

Multi View Image Fusion Using Ensemble Deep Learning Algorithm for Mri and CT Images

Adaptive spatial and frequency experts fusion network for medical image fusion

Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks

Multimodal medical image fusion using residual network 50 in non subsampled contourlet transform

Multi-modal medical image fusion using improved dual-channel PCNN

Edge-Enhanced Dilated Residual Attention Network for Multimodal Medical Image Fusion

Multi-modal medical image fusion based on densely-connected high-resolution CNN and hybrid transformer

Medical Image Segmentation Based on Multi-Modal Convolutional Neural Network: Study on Image Fusion Schemes