Polyamines and plant disease.

D. Walters

DOI: https://doi.org/10.1016/S0031-9422(03)00329-7

IF: 3.8

2003-09-01

Phytochemistry

Abstract:

What problem does this paper attempt to address?

Medical Image Segmentation Based on Multi-Modal Convolutional Neural Network: Study on Image Fusion Schemes

Zhe Guo,Xiang Li,Heng Huang,Ning Guo,Quanzheng Li

DOI: https://doi.org/10.1109/ISBI.2018.8363717

2017-11-03

Abstract:Image analysis using more than one modality (i.e. multi-modal) has been increasingly applied in the field of biomedical imaging. One of the challenges in performing the multimodal analysis is that there exist multiple schemes for fusing the information from different modalities, where such schemes are application-dependent and lack a unified framework to guide their designs. In this work we firstly propose a conceptual architecture for the image fusion schemes in supervised biomedical image analysis: fusing at the feature level, fusing at the classifier level, and fusing at the decision-making level. Further, motivated by the recent success in applying deep learning for natural image analysis, we implement the three image fusion schemes above based on the Convolutional Neural Network (CNN) with varied structures, and combined into a single framework. The proposed image segmentation framework is capable of analyzing the multi-modality images using different fusing schemes simultaneously. The framework is applied to detect the presence of soft tissue sarcoma from the combination of Magnetic Resonance Imaging (MRI), Computed Tomography (CT) and Positron Emission Tomography (PET) images. It is found from the results that while all the fusion schemes outperform the single-modality schemes, fusing at the feature level can generally achieve the best performance in terms of both accuracy and computational cost, but also suffers from the decreased robustness in the presence of large errors in any image modalities.

Computer Vision and Pattern Recognition,Machine Learning
Deep Learning-Based Image Segmentation on Multimodal Medical Imaging

Zhe Guo,Xiang Li,Heng Huang,Ning Guo,Quanzheng Li

DOI: https://doi.org/10.1109/trpms.2018.2890359

2019-03-01

IEEE Transactions on Radiation and Plasma Medical Sciences

Abstract:Multi-modality medical imaging techniques have been increasingly applied in clinical practice and research studies. Corresponding multi-modal image analysis and ensemble learning schemes have seen rapid growth and bring unique value to medical applications. Motivated by the recent success of applying deep learning methods to medical image processing, we first propose an algorithmic architecture for supervised multi-modal image analysis with cross-modality fusion at the feature learning level, classifier level, and decision-making level. We then design and implement an image segmentation system based on deep Convolutional Neural Networks (CNN) to contour the lesions of soft tissue sarcomas using multi-modal images, including those from Magnetic Resonance Imaging (MRI), Computed Tomography (CT) and Positron Emission Tomography (PET). The network trained with multi-modal images shows superior performance compared to networks trained with single-modal images. For the task of tumor segmentation, performing image fusion within the network (i.e. fusing at convolutional or fully connected layers) is generally better than fusing images at the network output (i.e. voting). This study provides empirical guidance for the design and application of multi-modal image analysis.
Multi-modal medical image fusion based on densely-connected high-resolution CNN and hybrid transformer

Quan Zhou,Shaozhuang Ye,Mingwei Wen,Zhiwen Huang,Mingyue Ding,Xuming Zhang

DOI: https://doi.org/10.1007/s00521-022-07635-1

2022-07-29

Neural Computing and Applications

Abstract:Multi-modal medical image fusion (MMIF) has found wide application in the field of disease diagnosis and surgical guidance. Despite the popularity of deep learning (DL)-based fusion methods, these DL algorithms cannot provide satisfactory fusion performance due to the difficulty in capturing the local information and the long-range dependencies effectively. To address these issues, this paper has presented an unsupervised MMIF method by combining a densely-connected high-resolution network (DHRNet) with a hybrid transformer. In this method, the local features are firstly extracted from the source image using the DHRNet. Then these features are input into the fine-grained attention module in the hybrid transformer to produce the global features by exploring their long-range dependencies. The local and global features are fused by the projection attention module in the hybrid transformer. Finally, based on the fused features, the fused result is reconstructed by the decoder network. The presented network is trained using an unsupervised loss function including edge preservation value, structural similarity, sum of the correlations of differences and structural tensor. Experiments on various multi-modal medical images show that, compared with several traditional and DL-based fusion methods, the presented method can generate visually better fused results and provide better quantitative metrics values.

computer science, artificial intelligence
MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer

Wenqing Wang,Ji He,Han Liu,Wei Yuan

DOI: https://doi.org/10.3390/s24134056

IF: 3.9

2024-06-22

Sensors

Abstract:The fusion of multi-modal medical images has great significance for comprehensive diagnosis and treatment. However, the large differences between the various modalities of medical images make multi-modal medical image fusion a great challenge. This paper proposes a novel multi-scale fusion network based on multi-dimensional dynamic convolution and residual hybrid transformer, which has better capability for feature extraction and context modeling and improves the fusion performance. Specifically, the proposed network exploits multi-dimensional dynamic convolution that introduces four attention mechanisms corresponding to four different dimensions of the convolutional kernel to extract more detailed information. Meanwhile, a residual hybrid transformer is designed, which activates more pixels to participate in the fusion process by channel attention, window attention, and overlapping cross attention, thereby strengthening the long-range dependence between different modes and enhancing the connection of global context information. A loss function, including perceptual loss and structural similarity loss, is designed, where the former enhances the visual reality and perceptual details of the fused image, and the latter enables the model to learn structural textures. The whole network adopts a multi-scale architecture and uses an unsupervised end-to-end method to realize multi-modal image fusion. Finally, our method is tested qualitatively and quantitatively on mainstream datasets. The fusion results indicate that our method achieves high scores in most quantitative indicators and satisfactory performance in visual qualitative analysis.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
SIMFusion: A Semantic Information-Guided Modality-Specific Fusion Network for MR Images

Xiaowen Zhang,Aiping Liu,Gang Yang,Yu Liu,Xun Chen

DOI: https://doi.org/10.1016/j.inffus.2024.102560

IF: 18.6

2024-01-01

Information Fusion

Abstract:Multi-modal medical image fusion aims to integrate distinct imaging modalities to yield more comprehensive and precise medical images, which can benefit the subsequent image analysis tasks. However, prevailing state-of-the-art image fusion methods, despite substantial advancements, do not explicitly address the efficient handling of complementary information between modalities. Moreover, most current multi-modal medical image fusion methods encounter challenges when integrating with practical tasks, which lack the guidance of semantic information, thus hindering the generation of high-quality images for accurate identification of lesion areas. To deal with these challenges, this paper introduces a novel semantic information-guided modality-specific fusion network for multi-modal magnetic resonance (MR) images, named SIMFusion. Specifically, we propose the decomposition branch to capture common and specific features from MR images of different modalities for fusion to reduce information redundancy through correlated mutual information loss. Subsequently, we obtain their semantic characteristics via the semantic branch utilizing a pre-train segmentation network and, finally, achieve an adaptive balance between the two sets of features through a specialized fusion strategy. Extensive experiments demonstrate the superiority of SIMFusion over existing competing techniques on both BraTS2019 and ISLES2022 datasets, with 8.1% improvement in MI and 38.7% in VIFF in T2-T1ce image pairs, highlighting the potential of the proposed method as a promising solution for MR image fusion in practical applications. Our code will be released at https://github.com/Zhangxw-ustc/SIMFusion.
An Attention-based Multi-Scale Feature Learning Network for Multimodal Medical Image Fusion

Meng Zhou,Xiaolan Xu,Yuxuan Zhang

DOI: https://doi.org/10.48550/arXiv.2212.04661

2022-12-09

Computer Vision and Pattern Recognition

Abstract:Medical images play an important role in clinical applications. Multimodal medical images could provide rich information about patients for physicians to diagnose. The image fusion technique is able to synthesize complementary information from multimodal images into a single image. This technique will prevent radiologists switch back and forth between different images and save lots of time in the diagnostic process. In this paper, we introduce a novel Dilated Residual Attention Network for the medical image fusion task. Our network is capable to extract multi-scale deep semantic features. Furthermore, we propose a novel fixed fusion strategy termed Softmax-based weighted strategy based on the Softmax weights and matrix nuclear norm. Extensive experiments show our proposed network and fusion strategy exceed the state-of-the-art performance compared with reference image fusion methods on four commonly used fusion metrics.
Sub-pixel multi-scale fusion network for medical image segmentation

Jing Li,Qiaohong Chen,Xian Fang

DOI: https://doi.org/10.1007/s11042-024-20338-0

IF: 2.577

2024-10-13

Multimedia Tools and Applications

Abstract:CNNs and Transformers have significantly advanced the domain of medical image segmentation. The integration of their strengths facilitates rich feature extraction but also introduces the challenge of mixed multi-scale feature fusion. To overcome this issue, we propose an innovative deep medical image segmentation framework termed Sub-pixel Multi-scale Fusion Network (SMFNet), which effectively incorporates the sub-pixel multi-scale feature fusion results of CNN and Transformer into the architecture. In particular, our design consists of three effective and practical modules. Primarily, we utilize the Sub-pixel Convolutional Module to synchronize the extracted features at multiple scales to a consistent resolution. In the next place, we develop the Three-level Enhancement Module to learn features from adjacent layers and perform information exchange. Lastly, we leverage the Hierarchical Adaptive Gate to fuse information from other contextual levels through the Sub-pixel Convolutional Module. Extensive experiments on the Synapse, ACDC, and ISIC 2018 datasets demonstrate the effectiveness of the proposed SMFNet, and our method is superior to other competitive CNN-based or Transformer-based segmentation methods.

computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering
Three-Dimensional Medical Image Fusion with Deformable Cross-Attention

Lin Liu,Xinxin Fan,Chulong Zhang,Jingjing Dai,Yaoqin Xie,Xiaokun Liang

2023-10-10

Abstract:Multimodal medical image fusion plays an instrumental role in several areas of medical image processing, particularly in disease recognition and tumor detection. Traditional fusion methods tend to process each modality independently before combining the features and reconstructing the fusion image. However, this approach often neglects the fundamental commonalities and disparities between multimodal information. Furthermore, the prevailing methodologies are largely confined to fusing two-dimensional (2D) medical image slices, leading to a lack of contextual supervision in the fusion images and subsequently, a decreased information yield for physicians relative to three-dimensional (3D) images. In this study, we introduce an innovative unsupervised feature mutual learning fusion network designed to rectify these limitations. Our approach incorporates a Deformable Cross Feature Blend (DCFB) module that facilitates the dual modalities in discerning their respective similarities and differences. We have applied our model to the fusion of 3D MRI and PET images obtained from 660 patients in the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset. Through the application of the DCFB module, our network generates high-quality MRI-PET fusion images. Experimental results demonstrate that our method surpasses traditional 2D image fusion methods in performance metrics such as Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM). Importantly, the capacity of our method to fuse 3D images enhances the information available to physicians and researchers, thus marking a significant step forward in the field. The code will soon be available online.

Image and Video Processing,Computer Vision and Pattern Recognition,Medical Physics
Multimodal medical image fusion based on interval gradients and convolutional neural networks

Xiaolong Gu,Ying Xia,Jie Zhang

DOI: https://doi.org/10.1186/s12880-024-01418-x

2024-09-05

Abstract:Many image fusion methods have been proposed to leverage the advantages of functional and anatomical images while compensating for their shortcomings. These methods integrate functional and anatomical images while presenting physiological and metabolic organ information, making their diagnostic efficiency far greater than that of single-modal images. Currently, most existing multimodal medical imaging fusion methods are based on multiscale transformation, which involves obtaining pyramid features through multiscale transformation. Low-resolution images are used to analyse approximate image features, and high-resolution images are used to analyse detailed image features. Different fusion rules are applied to achieve feature fusion at different scales. Although these fusion methods based on multiscale transformation can effectively achieve multimodal medical image fusion, much detailed information is lost during multiscale and inverse transformation, resulting in blurred edges and a loss of detail in the fusion images. A multimodal medical image fusion method based on interval gradients and convolutional neural networks is proposed to overcome this problem. First, this method uses interval gradients for image decomposition to obtain structure and texture images. Second, deep neural networks are used to extract perception images. Three methods are used to fuse structure, texture, and perception images. Last, the images are combined to obtain the final fusion image after colour transformation. Compared with the reference algorithms, the proposed method performs better in multiple objective indicators of Q EN , Q NIQE , Q SD , Q SSEQ and Q TMQI .
DDIFN: A Dual-discriminator Multi-modal Medical Image Fusion Network

Hui Liu,Shanshan Li,Jicheng Zhu,Kai Deng,Meng Liu,Liqiang Nie

DOI: https://doi.org/10.1145/3574136

2023-02-27

Abstract:Multi-modal medical image fusion is a long-standing important research topic that can obtain informative medical images and assist doctors diagnose and treat diseases more efficiently. However, most fusion methods extract and fuse features by subjectively defining constraints, which easily distorts the unique information of source images. In this work, we present a novel end-to-end unsupervised network to fuse multi-modal medical images. It is composed of a generator and two symmetrical discriminators. The former aims to generate a ”real-like” fused image based on a specifically designed content and structure loss, while the latter are devoted to distinguishing the differences between the fused image and the source ones. They are trained alternately until discriminators cannot distinguish the fused image from the source ones. In addition, the symmetrical discriminator scheme is conducive to maintaining the feature consistency among different modalities. More importantly, to enhance the retention degree of texture details, U-Net is adopted as the generator heuristically, where the up-sampling method is modified to bilinear interpolation for avoiding checkerboard artifacts. As for the optimization, we define the content loss function, which preserves the gradient information and pixel activity of source images. Both visual analysis and quantitative evaluation of experimental results show the superiority of our method as compared to the cutting-edge baselines.

computer science, information systems, theory & methods, software engineering
Multimodal medical image fusion and classification using deep learning techniques

D. Veeraiah,S. Sai Kumar,Rajendra Kumar Ganiya,Katta Subba Rao,J. Nageswara Rao,Ramaswamy Manjith,A. Rajaram

DOI: https://doi.org/10.3233/jifs-240018

2024-04-20

Journal of Intelligent & Fuzzy Systems

Abstract:Medical image fusion plays a crucial role in accurate medical diagnostics by combining images from various modalities. To address this need, we propose an AI model for efficient medical image fusion using multiple modalities. Our approach utilizes a Siamese convolutional neural network to construct a weight map based on pixel movement information extracted from multimodality medical images. We leverage medical picture pyramids to incorporate multiscale techniques, enhancing reliability beyond human visual intuition. Additionally, we dynamically adjust the fusion mode based on local comparisons of deconstructed coefficients. Evaluation metrics including F1-score, recall, accuracy, and precision are computed to assess performance, yielding impressive results: an F1-score of 0.8551 and a mutual information (MI) value of 2.8059. Experimental results demonstrate the superiority of our method, achieving a remarkable 99.61% accuracy in targeted experiments. Moreover, the Structural Similarity Index (SSIM) of our approach is 0.8551. Compared to state-of-the-art approaches, our model excels in medical picture classification, providing accurate diagnosis through high-quality fused images. This research advances medical image fusion techniques, offering a robust solution for precise medical diagnostics across various modalities.
Adaptive spatial and frequency experts fusion network for medical image fusion

Xianming Gu,Lihui Wang,Zeyu Deng,Ying Cao,Xingyu Huang,Yue-min Zhu

DOI: https://doi.org/10.1016/j.bspc.2024.106478

IF: 5.1

2024-06-02

Biomedical Signal Processing and Control

Abstract:Multi-modal medical image fusion is essential for the precise clinical diagnosis and surgical navigation since it can merge the complementary information in multi-modalities into a single image. Although existing deep learning-based fusion methods can fully exploit the semantic features of each modality, they cannot fuse them adaptively according to the importance of local and global features of each modality. To address this issue, we propose an adaptive spatial and frequency experts fusion network (ASFE-Fusion) for medical image fusion. Specifically, a cross attention spatial fusion (CASF) module is devised to adaptively fuse local features, which can constraint the consistency in spatial correlations by crossing the self-similarity map between two source images and therefore improve the details of the fused images. To complement the global information that may be lost in the CASF module, adaptive frequency fusion (AFF) module is introduced to fuse the features from a global perspective in frequency domain, which is useful for dealing with issue of intensity distortion. Subsequently, taking the fused features in spatial and frequency domains as two experts, an ensemble-learning-based spatial and frequency experts fusion (SFEF) module is used to adaptively fuse the local and global features. Finally, the fused image can be obtained from the fused features through a decoder by minimizing the content, gradient, and structure loss simultaneously. Extensive comparison experiments demonstrate that the proposed method outperforms state-of-the-art methods in terms of both visual quality and quantitative assessment. In addition, the downstream disease classification task also demonstrates the superiority of the proposed method.

engineering, biomedical
A Semantic-based Medical Image Fusion Approach

Fanda Fan,Yunyou Huang,Lei Wang,Xingwang Xiong,Zihan Jiang,Zhifei Zhang,Jianfeng Zhan

DOI: https://doi.org/10.48550/arXiv.1906.00225

2019-12-11

Abstract:It is necessary for clinicians to comprehensively analyze patient information from different sources. Medical image fusion is a promising approach to providing overall information from medical images of different modalities. However, existing medical image fusion approaches ignore the semantics of images, making the fused image difficult to understand. In this work, we propose a new evaluation index to measure the semantic loss of fused image, and put forward a Fusion W-Net (FW-Net) for multimodal medical image fusion. The experimental results are promising: the fused image generated by our approach greatly reduces the semantic information loss, and has better visual effects in contrast to five state-of-art approaches. Our approach and tool have great potential to be applied in the clinical setting.

Image and Video Processing,Computer Vision and Pattern Recognition
Multi-Level Medical Image Segmentation Network Based on Multi-Scale and Context Information Fusion Strategy

Dayu Tan,Zhiyuan Yao,Xin Peng,Haiping Ma,Yike Dai,Yansen Su,Weimin Zhong

DOI: https://doi.org/10.1109/tetci.2023.3306250

2024-01-01

IEEE Transactions on Emerging Topics in Computational Intelligence

Abstract:Accurate segmentation of human tissue structure from medical images is one of the critical links in medical image diagnosis. However, due to the medical image scale of different tissues varying significantly and being structurally complex, the low contrast between tissues and background in some medical imaging makes it challenging to identify. The existing models are difficult to extract representative multi-scale features of medical images that cannot accurately segment the organizational structure from the background in low-contrast medical images. To solve these problems, this study presents a scale and context information fusion network structure based on multi-type medical image segmentation (SCIF-Net), which contains three modules: feature pyramid encoder (FPE), multi-scale feature dynamic aggregation (MFDA), and adaptive spatial information fusion (ASIF). We build the FPE module to further enhance the representational ability of the network encoder output feature map at each stage. The MFDA module is used to effectively extract multi-scale information from the encoder output feature map and aggregate multi-scale features. The constructed ASIF module enables the network to selectively concentrate on the vital spatial information in the encoder feature map and merge the decoder feature map semantic information, minimizing background noise influence. Extensive experiments on the retinal segmentation task, gland segmentation task, and femur segmentation task, show that the SCIF-Net network outperforms other advanced methods.
Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks

Yu Liu,Yu Shi,Fuhao Mu,Juan Cheng,Chang Li,Xun Chen

DOI: https://doi.org/10.1109/tim.2022.3184360

IF: 5.6

2022-07-08

IEEE Transactions on Instrumentation and Measurement

Abstract:Medical image fusion aims to integrate the complementary information captured by images of different modalities into a more informative composite image. However, current study on medical image fusion suffers from several drawbacks: 1) existing methods are mostly designed for 2-D slice fusion, and they tend to lose spatial contextual information when fusing medical images with volumetric structure slice by slice individually; 2) the few existing 3-D medical image fusion methods fail in considering the characteristics of source modalities sufficiently, leading to the loss of important modality information; and 3) most existing works concentrate on pursuing good performance on visual perception and objective evaluation, while there is a severe lack of clinical problem-oriented study. In this article, to address these issues, we propose a multimodal MRI volumetric data fusion method based on an end-to-end convolutional neural network (CNN). In our network, an attention-based multimodal feature fusion (MMFF) module is presented for more effective feature learning. In addition, a specific loss function that considers the characteristics of different MRI modalities is designed to preserve the modality information. Experimental results demonstrate that the proposed method can obtain more competitive results on both visual quality and objective assessment when compared with some representative 3-D and 2-D medical image fusion methods. We further verify the significance of the proposed method for brain tumor segmentation by enriching the input modalities, and the results show that it is helpful to improve the segmentation accuracy. The source code of our fusion method is available at https://github.com/yuliu316316/3D-CNN-Fusion.

engineering, electrical & electronic,instruments & instrumentation
Multimodal Fusion for Enhanced Semantic Segmentation in Brain Tumor Imaging: Integrating Deep Learning and Guided Filtering Via Advanced 3D Semantic Segmentation Architectures

Abbadullah .H Saleh,Ümit Atila,Oğuzhan Menemencioğlu,Abbadullah .H Saleh

DOI: https://doi.org/10.1002/ima.23152

IF: 2.177

2024-08-13

International Journal of Imaging Systems and Technology

Abstract:ABSTRACT Brain tumor segmentation is paramount in medical diagnostics. This study presents a multistage segmentation model consisting of two main steps. First, the fusion of magnetic resonance imaging (MRI) modalities creates new and more effective tumor imaging modalities. Second, the semantic segmentation of the original and fused modalities, utilizing various modified architectures of the U‐Net model. In the first step, a residual network with multi‐scale backbone architecture (Res2Net) and guided filter are employed for pixel‐by‐pixel image fusion tasks without requiring any training or learning process. This method captures both detailed and base elements from the multimodal images to produce better and more informative fused images that significantly enhance the segmentation process. Many fusion scenarios were performed and analyzed, revealing that the best fusion results are attained when combining T2‐weighted (T2) with fluid‐attenuated inversion recovery (FLAIR) and T1‐weighted contrast‐enhanced (T1CE) with FLAIR modalities. In the second step, several models, including the U‐Net and its many modifications (adding attention layers, residual connections, and depthwise separable connections), are trained using both the original and fused modalities. Further, a “Model Selection‐based” fusion of these individual models is also considered for more enhancement. In the preprocessing step, the images are resized by cropping them to decrease the pixel count and minimize background interference. Experiments utilizing the brain tumor segmentation (BraTS) 2020 dataset were performed to verify the efficiency and accuracy of the proposed methodology. The “Model Selection‐based” fusion model achieved an average Dice score of 88.4%, an individual score of 91.1% for the whole tumor (WT) class, an average sensitivity score of 86.26%, and a specificity score of 91.7%. These results prove the robustness and high performance of the proposed methodology compared to other state‐of‐the‐art methods.

engineering, electrical & electronic,optics,imaging science & photographic technology
CASF-Net: Cross-attention and Cross-scale Fusion Network for Medical Image Segmentation

Jianwei Zheng,Hao Liu,Yuchao Feng,Jinshan Xu,Liang Zhao

DOI: https://doi.org/10.1016/j.cmpb.2022.107307

IF: 6.1

2022-12-13

Computer Methods and Programs in Biomedicine

Abstract:Background : Automatic segmentation of medical images has progressed greatly owing to the development of convolutional neural networks (CNNs). However, there are two uncertainties with current approaches based on convolutional operations: (1) how to eliminate the general limitations that CNNs lack the ability of modelling long-range dependencies and global contextual interactions, and (2) how to efficiently discover and integrate global and local features that are implied in the image. Notably, these two problems are interconnected, yet previous approaches mainly focus on the first problem and ignore the importance of information integration. Methods : In this paper, we propose a novel cross-attention and cross-scale fusion network (CASF-Net), which aims to explicitly tap the potential of dual-branch networks and fully integrate the coarse and fine-grained feature representations. Specifically, the well-designed dual-branch encoder hammers at modeling non-local dependencies and multi-scale contexts, significantly improving the quality of semantic segmentation. Moreover, the proposed cross-attention and cross-scale module efficiently perform multi-scale information fusion, being capable of further exploring the long-range contextual information. Results : Extensive experiments conducted on three different types of medical image segmentation tasks demonstrate the state-of-the-art performance of our proposed method both visually and numerically. Conclusions : This paper assembles the feature representation capabilities of CNN and transformer and proposes cross-attention and cross-scale fusion algorithms. The promising results show new possibilities of using cross-fusion mechanisms in more downstream medical image tasks.

engineering, biomedical,computer science, interdisciplinary applications,medical informatics, theory & methods
Feature extraction of multimodal medical image fusion using novel deep learning and contrast enhancement method

Jameel Ahmed Bhutto,Jiang Guosong,Ziaur Rahman,Muhammad Ishfaq,Zhengzheng Sun,Toufique Ahmed Soomro

DOI: https://doi.org/10.1007/s10489-024-05431-z

IF: 5.3

2024-05-02

Applied Intelligence

Abstract:The fusion of multimodal medical images has garnered painstaking attention for clinical diagnosis and surgical planning. Although various scholars have designed numerous fusion methods, the challenges of extracting substantial features without introducing noise and non-uniform contrast hindered the overall quality of fused photos. This paper presents a multimodal medical image fusion (MMIF) using a novel deep convolutional neural network (D-CNN) along with preprocessing schemes to circumvent the mentioned issues. A non-linear average median filtering (NL-AMF) and multiscale improved top-hat (MI-TH) approach are utilized at the preprocessing stage to remove noise and improve the contrast of images. The non-linear anisotropic diffusion (NL-AD) scheme is employed to split the photos into base and detailed parts. The fusion of base parts is accomplished by a dimension reduction method to retain the energy information. In contrast, the detailed parts are fused by novel D-CNN to preserve the enriched detailed features effectively. The simulation results demonstrate that the proposed method produces better brightness contrast and more image details than existing methods by acquiring 0.7649 to 0.8986 , 0.3520 to 0.4783, 0.7639 to 0.9056, 68.8932 to 81.0487 gain for quality transfer ratio from source photo to a generated photo ( ), feature mutual information (FMI), structural similarity index (SSIM), and average pixel intensity (API) respectively.

computer science, artificial intelligence
A multi-path adaptive fusion network for multimodal brain tumor segmentation

Yi Ding,Linpeng Gong,Mingfeng Zhang,Chang Li,Zhiguang Qin

DOI: https://doi.org/10.1016/j.neucom.2020.06.078

IF: 6

2020-10-01

Neurocomputing

Abstract:<p>The deep learning method has shown its outstanding performance in object recognition and becomes the first choice for medical image analysis. However, how to effectively propagate features in the learning layer and how to fuse low-level visual features and high-level semantic features, is still a challenging task. In addition, with the rapid development of the neural network, an increasing need of running Convolutional Neural Network (CNN) models with limited computing power and memory resource. To address these problems, this paper proposes a novel multi-path adaptive fusion network. More specifically, we apply the idea of "skip connection" in ResNets to the dense block so as to effectively reserve and propagate more low-level visual features. A contiguous memory mechanism has been realized by adopting direction connections from the state of preceding dense block to all layers of current dense block in the network. Then, a multi-path adaptive fusion dense block was adopted in the up-sampling process to adaptively adjust the low-level visual feature and then to fuse with high-level semantic features. By evaluating the proposed framework on the challenging BRATS2015 dataset, it can be proven that this framework achieves state-of-the-art results by comparing with other counterpart methods. Moreover, parameters of the proposed framework are much less than most published methods.</p>

computer science, artificial intelligence
Medical Image Fusion for Multiple Diseases Features Enhancement

Sajid Ullah Khan,Meshal Alharbi,Sajid Shah,Mohammed ELAffendi

DOI: https://doi.org/10.1002/ima.23197

IF: 2.177

2024-10-19

International Journal of Imaging Systems and Technology

Abstract:ABSTRACT Throughout the past 20 years, medical imaging has found extensive application in clinical diagnosis. Doctors may find it difficult to diagnose diseases using only one imaging modality. The main objective of multimodal medical image fusion (MMIF) is to improve both the accuracy and quality of clinical assessments by extracting structural and spectral information from source images. This study proposes a novel MMIF method to assist doctors and postoperations such as image segmentation, classification, and further surgical procedures. Initially, the intensity‐hue‐saturation (IHS) model is utilized to decompose the positron emission tomography (PET)/single photon emission computed tomography (SPECT) image, followed by a hue‐angle mapping method to discriminate high‐ and low‐activity regions in the PET images. Then, a proposed structure feature adjustment (SFA) mechanism is used as a fusion strategy for high‐ and low‐activity regions to obtain structural and anatomical details with minimum color distortion. In the second step, a new multi‐discriminator generative adversarial network (MDcGAN) approach is proposed for obtaining the final fused image. The qualitative and quantitative results demonstrate that the proposed method is superior to existing MMIF methods in preserving the structural, anatomical, and functional details of the PET/SPECT images. Through our assessment, involving visual analysis and subsequent verification using statistical metrics, it becomes evident that color changes contribute substantial visual information to the fusion of PET and MR images. The quantitative outcomes demonstrate that, in the majority of cases, the proposed algorithm consistently outperformed other methods. Yet, in a few instances, it achieved the second‐highest results. The validity of the proposed method was confirmed using diverse modalities, encompassing a total of 1012 image pairs.

engineering, electrical & electronic,optics,imaging science & photographic technology

Polyamines and plant disease.

Medical Image Segmentation Based on Multi-Modal Convolutional Neural Network: Study on Image Fusion Schemes

Deep Learning-Based Image Segmentation on Multimodal Medical Imaging

Multi-modal medical image fusion based on densely-connected high-resolution CNN and hybrid transformer

MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer

SIMFusion: A Semantic Information-Guided Modality-Specific Fusion Network for MR Images

An Attention-based Multi-Scale Feature Learning Network for Multimodal Medical Image Fusion

Sub-pixel multi-scale fusion network for medical image segmentation

Three-Dimensional Medical Image Fusion with Deformable Cross-Attention

Multimodal medical image fusion based on interval gradients and convolutional neural networks

DDIFN: A Dual-discriminator Multi-modal Medical Image Fusion Network

Multimodal medical image fusion and classification using deep learning techniques

Adaptive spatial and frequency experts fusion network for medical image fusion

A Semantic-based Medical Image Fusion Approach

Multi-Level Medical Image Segmentation Network Based on Multi-Scale and Context Information Fusion Strategy

Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks

Multimodal Fusion for Enhanced Semantic Segmentation in Brain Tumor Imaging: Integrating Deep Learning and Guided Filtering Via Advanced 3D Semantic Segmentation Architectures

CASF-Net: Cross-attention and Cross-scale Fusion Network for Medical Image Segmentation

Feature extraction of multimodal medical image fusion using novel deep learning and contrast enhancement method

A multi-path adaptive fusion network for multimodal brain tumor segmentation

Medical Image Fusion for Multiple Diseases Features Enhancement