Abstract:Background and objective: [18f]-fluorodeoxyglucose (fdg) positron emission tomography – computed tomography (pet-ct) is now the preferred imaging modality for staging many cancers. Pet images characterize tumoral glucose metabolism while ct depicts the complementary anatomical localization of the tumor. Automatic tumor segmentation is an important step in image analysis in computer aided diagnosis systems. Recently, fully convolutional networks (fcns), with their ability to leverage annotated datasets and extract image feature representations, have become the state-of-the-art in tumor segmentation. There are limited fcn based methods that support multi-modality images and current methods have primarily focused on the fusion of multi-modality image features at various stages, i.e., early-fusion where the multi-modality image features are fused prior to fcn, late-fusion with the resultant features fused and hyper-fusion where multi-modality image features are fused across multiple image feature scales. Early- and late-fusion methods, however, have inherent, limited freedom to fuse complementary multi-modality image features. The hyper-fusion methods learn different image features across different image feature scales that can result in inaccurate segmentations, in particular, in situations where the tumors have heterogeneous textures. Methods: we propose a recurrent fusion network (rfn), which consists of multiple recurrent fusion phases to progressively fuse the complementary multi-modality image features with intermediary segmentation results derived at individual recurrent fusion phases: (1) the recurrent fusion phases iteratively learn the image features and then refine the subsequent segmentation results; and, (2) the intermediary segmentation results allows our method to focus on learning the multi-modality image features around these intermediary segmentation results, which minimize the risk of inconsistent feature learning. Results: we evaluated our method on two pathologically proven non-small cell lung cancer pet-ct datasets. We compared our method to the commonly used fusion methods (early-fusion, late-fusion and hyper-fusion) and the state-of-the-art pet-ct tumor segmentation methods on various network backbones (resnet, densenet and 3d-unet). Our results show that the rfn provides more accurate segmentation compared to the existing methods and is generalizable to different datasets. Conclusions: we show that learning through multiple recurrent fusion phases allows the iterative re-use of multi-modality image features that refines tumor segmentation results. We also identify that our rfn produces consistent segmentation results across different network architectures.

SR2Net: A Separatable Reconstruction Residual Network Based on Channel Attention for PET and MRI Image Fusion

Image Fusion of CT and MR with Sparse Representation in NSST Domain

MBRARN: multibranch residual attention reconstruction network for medical image fusion

Direct Reconstruction and Separation for Triple-Tracer PET Imaging Based on Three-Dimensional Encoder-Decoder Network

Dual-tracer PET Image Direct Reconstruction and Separation Based on Three-Dimensional Encoder-Decoder Network

Fusion of Infrared and Visible Images Via Multi-Layer Convolutional Sparse Representation

MSAIF-Net: A Multistage Spatial Attention-Based Invertible Fusion Network for MR Images.

High-Quality Fusion and Visualization for MR-PET Brain Tumor Images via Multi-Dimensional Features

An Image Fusion Method Based on Sparse Representation and Sum Modified-Laplacian in NSCT Domain

MR-CT image fusion method of intracranial tumors based on Res2Net

CDBIFusion: A Cross-Domain Bidirectional Interaction Fusion Network for PET and MRI Images.

Three-Dimensional Medical Image Fusion with Deformable Cross-Attention

CIRF: Coupled Image Reconstruction and Fusion Strategy for Deep Learning Based Multi-Modal Image Fusion

Multi-Modal Medical Image Fusion Based on FusionNet in YIQ Color Space

Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks

LRFNet: A real-time medical image fusion method guided by detail information

Recurrent Feature Fusion Learning for Multi-Modality Pet-Ct Tumor Segmentation

Medical image fusion via discrete stationary wavelet transform and an enhanced radial basis function neural network

SIMFusion: A Semantic Information-Guided Modality-Specific Fusion Network for MR Images

RTFusion: A Multimodal Fusion Network with Significant Information Enhancement

Medical Image Fusion with Parameter-Adaptive Pulse Coupled Neural Network in Nonsubsampled Shearlet Transform Domain