Application and Prospect of Medical Image Fusion

Meng-qian WANG,Bo SHAO,Jian-hua YUE

DOI: https://doi.org/10.3969/j.issn.1674-1633.2007.05.021

2007-01-01

Abstract:Describes the ways of medical image fusion and two key techniques.Analyzes and compares various common used matching methods and several gorithms of intrinsic and extrinsic method.And forecasts the development trend of fusion.

What problem does this paper attempt to address?

Image Fusion of CT and MR with Sparse Representation in NSST Domain

Chenhui Qiu,Yuanyuan Wang,Huan Zhang,Shunren Xia

DOI: https://doi.org/10.1155/2017/9308745

IF: 2.809

2017-01-01

Computational and Mathematical Methods in Medicine

Abstract:Multimodal image fusion techniques can integrate the information from different medical images to get an informative image that is more suitable for joint diagnosis, preoperative planning, intraoperative guidance, and interventional treatment. Fusing images of CT and different MR modalities are studied in this paper. Firstly, the CT and MR images are both transformed to nonsubsampled shearlet transform (NSST) domain. So the low-frequency components and high-frequency components are obtained. Then the high-frequency components are merged using the absolute-maximum rule, while the low-frequency components are merged by a sparse representation- (SR-) based approach. And the dynamic group sparsity recovery (DGSR) algorithm is proposed to improve the performance of the SR-based approach. Finally, the fused image is obtained by performing the inverse NSST on the merged components. The proposed fusion method is tested on a number of clinical CT and MR images and compared with several popular image fusion methods. The experimental results demonstrate that the proposed fusion method can provide better fusion results in terms of subjective quality and objective evaluation.
Medical image fusion based on sparse representation and neighbor energy activity

Xiaosong Li,Weijun Wan,Fuqiang Zhou,Xiaoqi Cheng,Yuchan Jie,Haishu Tan

DOI: https://doi.org/10.1016/j.bspc.2022.104353

IF: 5.1

2022-11-14

Biomedical Signal Processing and Control

Abstract:Medical image fusion has become popular in recent years. The fused image can provide a more objective and comprehensive description of lesions and has significant clinical medical aid potential. In this paper, we propose a novel medical image fusion method based on sparse representation and neighbor energy activity that improves the quality of fused images and preserves key information in the source images, such as details, brightness, and color. The proposed method divides the source image into base and detail layers and adopts sparse representation to fuse the base layers. Further, a neighbor energy activity operator that effectively captures the changing features in the detail layers is utilized. The fused result is obtained by combining the selective layers. The proposed method is applicable to both grayscale and color image fusion problems. In experiments, ten sets of medical images were used as test images. The images included seven different diseases and one normal cranial image and covered five different fusion types: CT/T2, Gad/T2, PET/T1, PET/T2, and SPECT/T2. Further, it was compared with 11 state-of-the-art fusion algorithms, with six highly recognized metrics used for quantitative evaluation. The experimental results indicated that the proposed method outperformed several of the state-of-the-art methods in visual and objective evaluations. Additionally, in experiments conducted to medically analyze the fused images with eight different lesion conditions in the fused images, the fusion results were found to be practicable for medical assistance in actual clinics.

engineering, biomedical
Performance of Medical Image Fusion in High-level Analysis Tasks: A Mutual Enhancement Framework for Unaligned PAT and MRI Image Fusion

Yutian Zhong,Jinchuan He,Zhichao Liang,Shuangyang Zhang,Qianjin Feng,Wufan Chen,Li Qi

2024-07-04

Abstract:Photoacoustic tomography (PAT) offers optical contrast, whereas magnetic resonance imaging (MRI) excels in imaging soft tissue and organ anatomy. The fusion of PAT with MRI holds promising application prospects due to their complementary advantages. Existing image fusion have made considerable progress in pre-registered images, yet spatial deformations are difficult to avoid in medical imaging scenarios. More importantly, current algorithms focus on visual quality and statistical metrics, thus overlooking the requirements of high-level tasks. To address these challenges, we proposes a unsupervised fusion model, termed PAMRFuse+, which integrates image generation and registration. Specifically, a cross-modal style transfer network is introduced to simplify cross-modal registration to single-modal registration. Subsequently, a multi-level registration network is employed to predict displacement vector fields. Furthermore, a dual-branch feature decomposition fusion network is proposed to address the challenges of cross-modal feature modeling and decomposition by integrating modality-specific and modality-shared features. PAMRFuse+ achieves satisfactory results in registering and fusing unaligned PAT-MRI datasets. Moreover, for the first time, we evaluate the performance of medical image fusion with contour segmentation and multi-organ instance segmentation. Extensive experimental demonstrations reveal the advantages of PAMRFuse+ in improving the performance of medical image analysis tasks.

Image and Video Processing
Medical image fusion: A survey of the state of the art

Alex Pappachen James,Belur V. Dasarathy

DOI: https://doi.org/10.1016/j.inffus.2013.12.002

IF: 18.6

2014-09-01

Information Fusion

Abstract:Medical image fusion is the process of registering and combining multiple images from single or multiple imaging modalities to improve the imaging quality and reduce randomness and redundancy in order to increase the clinical applicability of medical images for diagnosis and assessment of medical problems. Multi-modal medical image fusion algorithms and devices have shown notable achievements in improving clinical accuracy of decisions based on medical images. This review article provides a factual listing of methods and summarizes the broad scientific challenges faced in the field of medical image fusion. We characterize the medical image fusion research based on (1) the widely used image fusion methods, (2) imaging modalities, and (3) imaging of organs that are under study. This review concludes that even though there exists several open ended technological and scientific challenges, the fusion of medical images has proved to be useful for advancing the clinical reliability of using medical imaging for medical diagnostics and analysis, and is a scientific discipline that has the potential to significantly grow in the coming years.

computer science, artificial intelligence, theory & methods
Medical image fusion based on improved multi-scale morphology gradient-weighted local energy and visual saliency map

Yi Zhang,Mingming Jin,Gang Huang

DOI: https://doi.org/10.1016/j.bspc.2022.103535

IF: 5.1

2022-04-01

Biomedical Signal Processing and Control

Abstract:Medical image fusion refers to the process of fusing images of different modes of the same object using image processing technology to maximize the mining of image information and improve image quality. At present, many methods are available to study image fusion, but they usually have shortcomings such as low image contrast and a weak ability to retain image details and edges. To solve these problems, we propose a new multimodal medical image fusion method. In this algorithm, the original image is decomposed into high-frequency and low-frequency information by non-subsampled shearlet transform (NSST). Low-frequency information is fused by visual saliency maps, which avoids edge loss caused by direct use of the coefficient maximum fusion rule. High-frequency information is fused by a method jointly guided by the improved multi-scale morphology gradient and weighted sum of eight-neighborhood-based modified Laplacian, which retains the texture details and the edge of the image. Finally, the fused image is generated by the NSST inverse transform. This strategy solves the problem of insufficient detail extraction in traditional algorithms, improves the overall appearance of the fused image, and enhances the contrast. To verify the effectiveness of the algorithm, we applied this technique to four different medical image modality combinations, compared the results with nine image fusion methods published in recent years, and evaluated the fused images using image quality evaluation indexes. Our algorithm achieved better results in terms of subjective vision and objective image quality evaluations and therefore should be competitive with existing technologies.

engineering, biomedical
A Review of Multimodal Medical Image Fusion Techniques

Bing Huang,Feng Yang,Mengxiao Yin,Xiaoying Mo,Cheng Zhong

DOI: https://doi.org/10.1155/2020/8279342

IF: 2.809

2020-04-23

Computational and Mathematical Methods in Medicine

Abstract:The medical image fusion is the process of coalescing multiple images from multiple imaging modalities to obtain a fused image with a large amount of information for increasing the clinical applicability of medical images. In this paper, we attempt to give an overview of multimodal medical image fusion methods, putting emphasis on the most recent advances in the domain based on (1) the current fusion methods, including based on deep learning, (2) imaging modalities of medical image fusion, and (3) performance analysis of medical image fusion on mainly data set. Finally, the conclusion of this paper is that the current multimodal medical image fusion research results are more significant and the development trend is on the rise but with many challenges in the research field.

mathematical & computational biology
Application of Medical Image Fusion in Radiation Therapy for Cancer

鹿红,李兵,朱锡旭

DOI: https://doi.org/10.3969/j.issn.1672-4992.2009.08.074

2009-01-01

Journal of Modern Oncology

Abstract:The technique of medical image fusion is being investigated in radiation therapy,especially in modern accurate radiotherapy.The classification,method and procedure of image fusion were introduced and different imaging modality applications in radiotherapy were reviewed.It is obvious that medical image fusion will play an important role in biologically conformal radiotherapy.
Recent Advances in Sparse Representation Based Medical Image Fusion

Yu Liu,Xun Chen,Aiping Liu,Rabab K. Ward,Z. Jane Wang

DOI: https://doi.org/10.1109/mim.2021.9400960

2021-01-01

Abstract:Medical image fusion, which aims to combine multi-source information captured by different imaging modalities, is of great significance to medical professionals for precise diagnosis and treatment. In the last decade, sparse representation (SR)-based approach has emerged as a very active direction in the field of medical image fusion, due to its powerful ability for image representation. In this paper, we mainly present an overview of the recent advances achieved in SR-based medical image fusion, ranging from the conventional local and single-component SR-based methods to the latest global and multi-component SR-based methods. In addition, several major challenges remained in this direction are presented and some future prospects are discussed.
[Advances in the diagnosis of prostate cancer based on image fusion]

Wenbin Luo,Pei Wang,Yiwei Zhang,Gengqiang Shi

DOI: https://doi.org/10.7507/1001-5515.202403054

2024-10-25

Abstract:Image fusion currently plays an important role in the diagnosis of prostate cancer (PCa). Selecting and developing a good image fusion algorithm is the core task of achieving image fusion, which determines whether the fusion image obtained is of good quality and can meet the actual needs of clinical application. In recent years, it has become one of the research hotspots of medical image fusion. In order to make a comprehensive study on the methods of medical image fusion, this paper reviewed the relevant literature published at home and abroad in recent years. Image fusion technologies were classified, and image fusion algorithms were divided into traditional fusion algorithms and deep learning (DL) fusion algorithms. The principles and workflow of some algorithms were analyzed and compared, their advantages and disadvantages were summarized, and relevant medical image data sets were introduced. Finally, the future development trend of medical image fusion algorithm was prospected, and the development direction of medical image fusion technology for the diagnosis of prostate cancer and other major diseases was pointed out.
A Medical Image Fusion Method Based on Convolutional Neural Networks

Yu Liu,Xun Chen,Juan Cheng,Hu Peng

DOI: https://doi.org/10.23919/icif.2017.8009769

2017-01-01

Abstract:Medical image fusion technique plays an an increasingly critical role in many clinical applications by deriving the complementary information from medical images with different modalities. In this paper, a medical image fusion method based on convolutional neural networks (CNNs) is proposed. In our method, a siamese convolutional network is adopted to generate a weight map which integrates the pixel activity information from two source images. The fusion process is conducted in a multi-scale manner via image pyramids to be more consistent with human visual perception. In addition, a local similarity based strategy is applied to adaptively adjust the fusion mode for the decomposed coefficients. Experimental results demonstrate that the proposed method can achieve promising results in terms of both visual quality and objective assessment.
Medical Image Fusion via an Effective Wavelet-Based Approach

Yong Yang,Dong Sun Park,Shuying Huang,Nini Rao

DOI: https://doi.org/10.1155/2010/579341

2010-01-01

EURASIP Journal on Advances in Signal Processing

Abstract:A novel wavelet-based approach for medical image fusion is presented, which is developed by taking into not only account the characteristics of human visual system (HVS) but also the physical meaning of the wavelet coefficients. After the medical images to be fused are decomposed by the wavelet transform, different-fusion schemes for combining the coefficients are proposed: coefficients in low-frequency band are selected with a visibility-based scheme, and coefficients in high-frequency bands are selected with a variance based method. To overcome the presence of noise and guarantee the homogeneity of the fused image, all the coefficients are subsequently performed by a window-based consistency verification process. The fused image is finally constructed by the inverse wavelet transform with all composite coefficients. To quantitatively evaluate and prove the performance of the proposed method, series of experiments and comparisons with some existing fusion methods are carried out in the paper. Experimental results on simulated and real medical images indicate that the proposed method is effective and can get satisfactory fusion results.
Adaptive Decomposition Method for Multi-Modal Medical Image Fusion.

Jing Wang,Xiongfei Li,Yan Zhang,Xiaoli Zhang

DOI: https://doi.org/10.1049/iet-ipr.2017.1067

IF: 2.3

2018-01-01

IET Image Processing

Abstract:In traditional image fusion, source images are separated into a fixed space. The low-frequency part and the high-frequency part are not discriminated according to the nature of the image. Traditional fusion rules often use a fixed proportion, causing colour distortion. In this study, a new adaptive decomposition algorithm is proposed to distinguish high frequency and low frequency of structure image to obtain smoothing layer and texture layer. The smoothing layer of the structural image and the colour information of the function image are fused according to dynamic rules, and then the texture layer is added. On the basis of the objective evaluation metrics, the spectral information evaluation metrics are introduced to evaluate the retention of colour. In the experiments, the proposed method is compared with other six classical image fusion methods. The experiment results show that the proposed method can retain the colour information and structure information very well at the same time. Concerning subjective and objective evaluation, the proposed algorithm is superior to other algorithms.
Medical image fusion using multi-level local extrema

Zhiping Xu

DOI: https://doi.org/10.1016/j.inffus.2013.01.001

IF: 18.6

2014-01-01

Information Fusion

Abstract:The fusion of data for medical imaging has become a central issue in such biomedical applications as image-guided surgery and radiotherapy. The multi-level local extrema (MLE) representation has been shown to have many advantages over conventional image representation methods. In this paper, we propose a new fusion algorithm for multi-modal medical images based on MLE. Our method enables the decomposition of input images into coarse and detailed layers in the MLE schema, and utilizes local energy and contrast fusion rules for coefficient selection in the different layers. This preserves more detail in the source images and further improves the quality of the fused image. The final fused image is obtained from the superposition of selected coefficients in the coarse and detailed layers. We illustrate the performance of the proposed method using three groups of medical images from different sources as our experimental subjects. We also compare our method with other techniques using cumulative mutual information, the objective image fusion performance measure, spatial frequency, and a blind quality index. Experimental results show that our method achieves a superior performance in both subjective and objective assessment criteria.
Research Status of Image Fusion: A Review

Xinman Zhang,Jiayu Zhang,Jie Kou,Xuebin Xu

DOI: https://doi.org/10.1109/crc.2019.00044

2019-01-01

Abstract:With the development of computer technology, sensor technology and information processing technology, image fusion, as a powerful tool of information fusion, has been widely used in automatic target recognition, military and medical image fields. This technology utilizes the complementarity of information advantages contained in each image to obtain a clear, comprehensive and accurate image description of the same scene through a certain algorithm. In this paper, we firstly summarized the research status of image fusion methods comprehensively. Secondly, the existing fusion methods of homogeneity and heterogeneity are summarized. Thirdly, several evaluation indexes are introduced. Finally, this paper points out the current problems of image fusion. In a word, the research of image fusion will still have a great development in the next few years.
Fusion in medical imaging: theory, interests and industrial applications

V Barra,P Briandet,J Y Boire

Abstract:The accumulation of several data coming from medical images and signals, expert knowledge and databases is becoming very common for the study of a given pathology. The aggregation of all this information is mentally performed by a clinician, and generally allows for a better medical decision in clinical studies. We propose in this article a fusion method that models this aggregation process. This method is a three step scheme, that first transforms all the available information in a common theoretical frame, then aggregates these data using their redundancy and their conflicts, and finally computes a new information synthesizing all the initial knowledge. We first introduce in this article the fusion scheme and its theoretical aspects, and we particularly focus on the three steps of the process. We then detail the software implementation of this concept, achieved in collaboration with SEGAMI Corporation Inc. We finally apply this concept to a real clinical problem, the study of Alzheimer's disease using MR and SPECT images, and we show very encouraging preliminary results.
An Integrated Dictionary-Learning Entropy-Based Medical Image Fusion Framework

Guanqiu Qi,Jinchuan Wang,Qiong Zhang,Fancheng Zeng,Zhiqin Zhu

DOI: https://doi.org/10.3390/fi9040061

2017-10-06

Future Internet

Abstract:Image fusion is widely used in different areas and can integrate complementary and relevant information of source images captured by multiple sensors into a unitary synthetic image. Medical image fusion, as an important image fusion application, can extract the details of multiple images from different imaging modalities and combine them into an image that contains complete and non-redundant information for increasing the accuracy of medical diagnosis and assessment. The quality of the fused image directly affects medical diagnosis and assessment. However, existing solutions have some drawbacks in contrast, sharpness, brightness, blur and details. This paper proposes an integrated dictionary-learning and entropy-based medical image-fusion framework that consists of three steps. First, the input image information is decomposed into low-frequency and high-frequency components by using a Gaussian filter. Second, low-frequency components are fused by weighted average algorithm and high-frequency components are fused by the dictionary-learning based algorithm. In the dictionary-learning process of high-frequency components, an entropy-based algorithm is used for informative blocks selection. Third, the fused low-frequency and high-frequency components are combined to obtain the final fusion results. The results and analyses of comparative experiments demonstrate that the proposed medical image fusion framework has better performance than existing solutions.
Medical image fusion based on quasi-cross bilateral filtering

Yi Zhang,Meng Wang,Xunpeng Xia,Dandan Sun,Xinhong Zhou,Yao Wang,Qian Dai,Mingming Jin,Liu Liu,Gang Huang

DOI: https://doi.org/10.1016/j.bspc.2022.104259

IF: 5.1

2023-02-01

Biomedical Signal Processing and Control

Abstract:Image fusion technology is a method that uses a specific algorithm to extract and fuse features from multiple images and then combine them into one image. Unfortunately, most of the current image fusion methods can't balance the fusion performance, time consumption and model complexity. Firstly, in the classical bilateral filtering algorithm, the pixel domain originally had only the pixel value of a single image, and the unique information of the image relative to another source image can not be obtained, resulting in partial loss of the lesion information in the fused image. Based on the above problems, an image decomposition method based on quasi-cross bilateral filtering (QBF) is proposed. The image is decomposed into an energy layer with pure intensity information and a structure layer with rich details. Then, the visual saliency detection map (VSDM) is used to guide the fusion of the energy layers to take full advantage of the edge contour extraction to improve the edge contour sharpness of the fused image and to retain the image detail information fully. The improved multi-level morphology gradient (IMLMG) combined with the weighted sum of eight-neighborhood-based modified Laplacian (WSEML) was used to guide the fusion of the structure layer, which effectively improved the sharpness of the contour edges of the fused image. Finally, to evaluate the performance of the proposed fusion method, we fused six groups of medical images using the proposed method, and then evaluated the fusion quality by subjective vision and objective quality evaluation. The results showed that the proposed method performed better than nine other methods in terms of providing detailed information, edge contour, and overall contrast.

engineering, biomedical
Deep learning methods for medical image fusion: A review

Tao Zhou,QianRu Cheng,HuiLing Lu,Qi Li,XiangXiang Zhang,Shi Qiu

DOI: https://doi.org/10.1016/j.compbiomed.2023.106959

IF: 7.7

2023-04-30

Computers in Biology and Medicine

Abstract:The image fusion methods based on deep learning has become a research hotspot in the field of computer vision in recent years. This paper reviews these methods from five aspects: Firstly, the principle and advantages of image fusion methods based on deep learning are expounded; Secondly, the image fusion methods are summarized in two aspects: End-to-End and Non-End-to-End, according to the different tasks of deep learning in the feature processing stage, the non-end-to-end image fusion methods are divided into two categories: deep learning for decision mapping and deep learning for feature extraction. According to the different types of the networks, the end-to-end image fusion methods are divided into three categories: image fusion methods based on Convolutional Neural Network, Generative Adversarial Network, and Encoder-Decoder Network; Thirdly, the application of the image fusion methods based on deep learning in medical image field is summarized from two aspects: method and data set; Fourthly, evaluation metrics commonly used in the field of medical image fusion are sorted out from 14 aspects; Fifthly, the main challenges faced by the medical image fusion are discussed from two aspects: data sets and fusion methods. And the future development direction is prospected. This paper systematically summarizes the image fusion methods based on the deep learning, which has a positive guiding significance for the in-depth study of multi modal medical images.

engineering, biomedical,computer science, interdisciplinary applications,mathematical & computational biology,biology
Medical image fusion via discrete stationary wavelet transform and an enhanced radial basis function neural network

Zhen Chao,Xingguang Duan,Shuangfu Jia,Xuejun Guo,Hao Liu,Fucang Jia

DOI: https://doi.org/10.1016/j.asoc.2022.108542

IF: 8.7

2022-03-01

Applied Soft Computing

Abstract:Medical image fusion of images obtained via different modes can expand the inherent information of original images, whereby the fused image has a superior ability to display details than the original sub-images, to facilitate diagnosis and treatment selection. In medical image fusion, an inherent challenge is to effectively combine the most useful information and image details without information loss. Despite the many methods that have been proposed, the effective retention and presentation of information proves challenging. Therefore, we proposed and evaluated a novel image fusion method based on the discrete stationary wavelet transform (DSWT) and radial basis function neural network (RBFNN). First, we analyze the details or feature information of two images to be processed by DSWT by using two-level decomposition to separate each image into seven parts, comprising both high-frequency and low-frequency sub-bands. Considering the gradient and energy attributes of the target, we substituted the pending parts in the same position in the two images by using the proposed enhanced RBFNN. The input, hidden, and output layers of the neural network comprised 8, 40, and 1 neuron(s), respectively. From the seven neural networks, we obtained seven fused parts. Finally, through inverse wavelet transform, we obtained the final fused image. For the neural network training method, the hybrid adaptive gradient descent algorithm (AGDA) and gravitational search algorithm (GSA) were implemented. The final experimental results revealed that the novel method has significantly better performance than the current state-of-the-art methods.

computer science, artificial intelligence, interdisciplinary applications
Study of Image Fusion and Evaluation Methods

Ke Wang

2007-01-01

Abstract:The paper introduces the basic concepts and different levels of image fusion technology and elaborates the fundamental principles and methods employed in pixel-level image fusion technology.Then it generalizes the objective evaluation criteria of fused image,and compares the experimental results with the criteria.Finally,the prospect of the development of image fusion is described.

Application and Prospect of Medical Image Fusion

Image Fusion of CT and MR with Sparse Representation in NSST Domain

Medical image fusion based on sparse representation and neighbor energy activity

Performance of Medical Image Fusion in High-level Analysis Tasks: A Mutual Enhancement Framework for Unaligned PAT and MRI Image Fusion

Medical image fusion: A survey of the state of the art

Medical image fusion based on improved multi-scale morphology gradient-weighted local energy and visual saliency map

A Review of Multimodal Medical Image Fusion Techniques

Application of Medical Image Fusion in Radiation Therapy for Cancer

Recent Advances in Sparse Representation Based Medical Image Fusion

[Advances in the diagnosis of prostate cancer based on image fusion]

A Medical Image Fusion Method Based on Convolutional Neural Networks

Medical Image Fusion via an Effective Wavelet-Based Approach

Adaptive Decomposition Method for Multi-Modal Medical Image Fusion.

Medical image fusion using multi-level local extrema

Research Status of Image Fusion: A Review

Fusion in medical imaging: theory, interests and industrial applications

An Integrated Dictionary-Learning Entropy-Based Medical Image Fusion Framework

Medical image fusion based on quasi-cross bilateral filtering

Deep learning methods for medical image fusion: A review

Medical image fusion via discrete stationary wavelet transform and an enhanced radial basis function neural network

Study of Image Fusion and Evaluation Methods