Abstract:Programmed death-ligand 1 (PD-L1) expressions play a crucial role in guiding therapeutic interventions such as the use of tyrosine kinase inhibitors (TKIs) and immune checkpoint inhibitors (ICIs) in lung cancer. Conventional determination of PD-L1 status includes careful surgical or biopsied tumor specimens. These specimens are gathered through invasive procedures, representing a risk of difficulties and potential challenges in getting reliable and representative tissue samples. Using a single center cohort of 189 patients, our objective was to evaluate various fusion methods that used non-invasive computed tomography (CT) and F-FDG positron emission tomography (PET) images as inputs to various deep learning models to automatically predict PD-L1 in non-small cell lung cancer (NSCLC). We compared three different architectures (ResNet, DenseNet, and EfficientNet) and considered different input data (CT only, PET only, PET/CT early fusion, PET/CT late fusion without as well as with partially and fully shared weights to determine the best model performance. Models were assessed utilizing areas under the receiver operating characteristic curves (AUCs) considering their 95% confidence intervals (CI). The fusion of PET and CT images as input yielded better performance for PD-L1 classification. The different data fusion schemes systematically outperformed their individual counterparts when used as input of the various deep models. Furthermore, early fusion consistently outperformed late fusion, probably as a result of its capacity to capture more complicated patterns by merging PET and CT derived content at a lower level. When we looked more closely at the effects of weight sharing in late fusion architectures, we discovered that while it might boost model stability, it did not always result in better results. This suggests that although weight sharing could be beneficial when modality parameters are similar, the anatomical and metabolic information provided by CT and PET scans are too dissimilar to consistently lead to improved PD-L1 status predictions.

Multimodal Classification on PET/CT Image Fusion for Lung Cancer: A Comprehensive Survey

Co-Learning Feature Fusion Maps from PET-CT Images of Lung Cancer

Lung Cancer Diagnosis Based on Image Fusion and Prediction Using CT and PET Image

Optimized pixel level image fusion for lung cancer detection over MRI and pet image

Multi-modal Medical Image Fusion For Non-Small Cell Lung Cancer Classification

Multimodality Attention-Guided 3-D Detection of Nonsmall Cell Lung Cancer in 18 F-FDG PET/CT Images

Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection

Multimodal medical image fusion review: Theoretical background and recent advances

Medical image fusion: A survey of the state of the art

Multimodal medical image fusion and classification using deep learning techniques

A Systematic Literature Review on Multimodal Medical Image Fusion

NSCT-DCT based Fourier Analysis for Fusion of Multimodal Images

Medical Image Fusion for Multiple Diseases Features Enhancement

Fine-Grained Lung Cancer Classification from PET and CT Images Based on Multidimensional Attention Mechanism

Automated PD-L1 status prediction in lung cancer with multi-modal PET/CT fusion

Multimodal Medical Image Fusion Techniques – A Review

Multimodal medical image fusion algorithm in the era of big data

Advanced Hybrid Attention-Based Deep Learning Network with Heuristic Algorithm for Adaptive CT and PET Image Fusion in Lung Cancer Detection

A review of deep learning-based information fusion techniques for multimodal medical image classification

A Review of Multimodal Medical Image Fusion Techniques

High-Quality Fusion and Visualization for MR-PET Brain Tumor Images via Multi-Dimensional Features