Abstract:With the advance of medical imaging technologies, multimodal images such as magnetic resonance images (MRI) and positron emission tomography (PET) can capture subtle structural and functional changes of brain, facilitating the diagnosis of brain diseases such as Alzheimer's disease (AD). In practice, multimodal images may be incomplete since PET is often missing due to high financial costs or availability. Most of the existing methods simply excluded subjects with missing data, which unfortunately reduced the sample size. In addition, how to extract and combine multimodal features is still challenging. To address these problems, we propose a deep learning framework to integrate a task-induced pyramid and attention generative adversarial network (TPA-GAN) with a pathwise transfer dense convolution network (PT-DCN) for imputation and classification of multimodal brain images. First, we propose a TPA-GAN to integrate pyramid convolution and attention module as well as disease classification task into GAN for generating the missing PET data with their MRI. Then, with the imputed multimodal images, we build a dense convolution network with pathwise transfer blocks to gradually learn and combine multimodal features for final disease classification. Experiments are performed on ADNI-1/2 datasets to evaluate our method, achieving superior performance in image imputation and brain disease diagnosis compared to state-of-the-art methods.

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly focus on two aspects: 1. **The problem of incomplete multi - modal brain image data**: In practical applications, due to the high cost of obtaining PET (Positron Emission Tomography) images or the limited availability of equipment, PET images are often missing in multi - modal brain image data. Most of the existing methods simply exclude samples with missing data, which not only reduces the sample size but also may lead to information loss and affect the accuracy of disease diagnosis. 2. **How to effectively extract and fuse multi - modal features**: Extracting features from different imaging modalities (such as MRI and PET) and effectively combining these features for disease classification is still a challenge. Data from different modalities provide complementary information, but how to efficiently use this information for disease diagnosis, especially for the early diagnosis of Alzheimer's disease (AD) and mild cognitive impairment (MCI), is an urgent problem to be solved. To solve the above problems, the author proposes a deep - learning framework, which consists of two main parts: - **Task - Induced Pyramid and Attention Generative Adversarial Network (TPA - GAN)**: It is used to generate missing PET images. TPA - GAN generates missing PET images corresponding to MRI by integrating pyramid convolution and attention modules, as well as disease classification tasks. This part aims to improve the quality of generated images, especially to preserve abnormal changes related to diseases. - **Path - Transfer Dense Convolutional Network (PT - DCN)**: It is used for the classification of multi - modal brain images. PT - DCN finally realizes the classification of brain diseases by gradually learning and fusing multi - modal features. This network designs path - transfer blocks to make full use of the complementary information between different - modality images, thereby improving the classification performance. Through the combination of these two parts, this study aims to provide an effective solution that can not only deal with the missing problem in multi - modal brain image data but also efficiently extract and fuse multi - modal features for the early diagnosis of Alzheimer's disease and the transformation prediction of mild cognitive impairment.

Task-Induced Pyramid and Attention GAN for Multimodal Brain Image Imputation and Classification in Alzheimer's Disease

Pyramid-attentive GAN for Multimodal Brain Image Complementation in Alzheimer’s Disease Classification

Multimodal transformer network for incomplete image generation and diagnosis of Alzheimer's disease

A coupled-GAN architecture to fuse MRI and PET image features for multi-stage classification of Alzheimer's disease

Bidirectional Mapping of Brain MRI and PET With 3D Reversible GAN for the Diagnosis of Alzheimer’s Disease

Cross-modal Medical Image Generation Based on Pyramid Convolutional Attention Network

MRI to PET Cross-Modality Translation using Globally and Locally Aware GAN (GLA-GAN) for Multi-Modal Diagnosis of Alzheimer's Disease

Cross-Modality Generation of Amyloid PET from FDG PET for Alzheimer’s Disease Diagnosis

Early diagnosis of Alzheimer's disease using dual GAN model with pyramid attention networks

A Hybrid Deep Learning Method for Early and Late Mild Cognitive Impairment Diagnosis With Incomplete Multimodal Data

Tensorizing GAN with High-Order Pooling for Alzheimer's Disease Assessment

Brain Status Transferring Generative Adversarial Network for Decoding Individualized Atrophy in Alzheimer's Disease

Alzheimer's Disease Classification Accuracy is Improved by MRI Harmonization based on Attention-Guided Generative Adversarial Networks

Alzheimer's Disease Prediction via Brain Structural-Functional Deep Fusing Network

Enhancing magnetic resonance imaging-driven Alzheimer’s disease classification performance using generative adversarial learning

An Effective Multimodal Image Fusion Method Using MRI and PET for Alzheimer's Disease Diagnosis

Diagnosis of Alzheimer's Disease via Multi-modality 3D Convolutional Neural Network

GANDALF: Generative Adversarial Networks with Discriminator-Adaptive Loss Fine-tuning for Alzheimer's Disease Diagnosis from MRI

Cross-Modal Transformer GAN: A Brain Structure-Function Deep Fusing Framework for Alzheimer's Disease

Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis

Adversarial Learning for MRI Reconstruction and Classification of Cognitively Impaired Individuals