Abstract:Background and objective: [18f]-fluorodeoxyglucose (fdg) positron emission tomography – computed tomography (pet-ct) is now the preferred imaging modality for staging many cancers. Pet images characterize tumoral glucose metabolism while ct depicts the complementary anatomical localization of the tumor. Automatic tumor segmentation is an important step in image analysis in computer aided diagnosis systems. Recently, fully convolutional networks (fcns), with their ability to leverage annotated datasets and extract image feature representations, have become the state-of-the-art in tumor segmentation. There are limited fcn based methods that support multi-modality images and current methods have primarily focused on the fusion of multi-modality image features at various stages, i.e., early-fusion where the multi-modality image features are fused prior to fcn, late-fusion with the resultant features fused and hyper-fusion where multi-modality image features are fused across multiple image feature scales. Early- and late-fusion methods, however, have inherent, limited freedom to fuse complementary multi-modality image features. The hyper-fusion methods learn different image features across different image feature scales that can result in inaccurate segmentations, in particular, in situations where the tumors have heterogeneous textures. Methods: we propose a recurrent fusion network (rfn), which consists of multiple recurrent fusion phases to progressively fuse the complementary multi-modality image features with intermediary segmentation results derived at individual recurrent fusion phases: (1) the recurrent fusion phases iteratively learn the image features and then refine the subsequent segmentation results; and, (2) the intermediary segmentation results allows our method to focus on learning the multi-modality image features around these intermediary segmentation results, which minimize the risk of inconsistent feature learning. Results: we evaluated our method on two pathologically proven non-small cell lung cancer pet-ct datasets. We compared our method to the commonly used fusion methods (early-fusion, late-fusion and hyper-fusion) and the state-of-the-art pet-ct tumor segmentation methods on various network backbones (resnet, densenet and 3d-unet). Our results show that the rfn provides more accurate segmentation compared to the existing methods and is generalizable to different datasets. Conclusions: we show that learning through multiple recurrent fusion phases allows the iterative re-use of multi-modality image features that refines tumor segmentation results. We also identify that our rfn produces consistent segmentation results across different network architectures.

The value of a neural network based on multi-scale feature fusion to ultrasound images for the differentiation in thyroid follicular neoplasms

Ovarian Ultrasound Image Segmentation Algorithm with Fused Multi-Scale Features

Automatic Detection of Focal Liver Lesions in Multi-phase CT Images Using A Multi-channel & Multi-scale CNN.

Self-supervised multi-modal fusion network for multi-modal thyroid ultrasound image diagnosis

Thyroid nodule recognition using a joint convolutional neural network with information fusion of ultrasound images and radiofrequency data

A Multimodal Deep Learning Model for Preoperative Risk Prediction of Follicular Thyroid Carcinoma

Deep learning based on ultrasound to differentiate pathologically proven atypical and typical medullary thyroid carcinoma from follicular thyroid adenoma

Breast Tumor Classification in Ultrasound Images by Fusion of Deep Convolutional Neural Network and Shallow LBP Feature

FMRNet: A fused network of multiple tumoral regions for breast tumor classification with ultrasound images

A pre-trained convolutional neural network based method for thyroid nodule diagnosis.

FDE-net: Frequency-domain enhancement network using dynamic-scale dilated convolution for thyroid nodule segmentation

Multimodal Diverse Granularity Fusion Network based on US and CT Images for Lymph Node Metastasis Prediction of Thyroid Carcinoma

Using Deep Convolutional Neural Networks for Enhanced Ultrasonographic Image Diagnosis of Differentiated Thyroid Cancer

MFA-Net: Multi-Scale feature fusion attention network for liver tumor segmentation

A generic deep learning framework to classify thyroid and breast lesions in ultrasound images

Expansive Receptive Field and Local Feature Extraction Network: Advancing Multiscale Feature Fusion for Breast Fibroadenoma Segmentation in Sonography

A multi-channel deep convolutional neural network for multi-classifying thyroid diseases

Multi-Level fusion graph neural network: Application to PET and CT imaging for risk stratification of head and neck cancer

Recurrent Feature Fusion Learning for Multi-Modality Pet-Ct Tumor Segmentation

Multi-path synergic fusion deep neural network framework for breast mass classification using digital breast tomosynthesis

Federated learning via multi-attention guided UNet for thyroid nodule segmentation of ultrasound images