Abstract:<section class="article-section__content"><h3 class="article-section__sub-title section1"> Purpose</h3><p>Deep learning models have had a great success in disease classifications using large data pools of skin cancer images or lung X‐rays. However, data scarcity has been the roadblock of applying deep learning models directly on prostate multiparametric MRI (mpMRI). Although model interpretation has been heavily studied for natural images for the past few years, there has been a lack of interpretation of deep learning models trained on medical images.</p><p>In this paper, an efficient convolutional neural network (CNN) was developed and the model interpretation at various convolutional layers was systematically analyzed to improve the understanding of how CNN interprets multimodality medical images and the predictive powers of features at each layer. The problem of small sample size was addressed by feeding the intermediate features into a traditional classification algorithm known as weighted extreme learning machine (wELM), with imbalanced distribution among output categories taken into consideration.</p></section><section class="article-section__content"><h3 class="article-section__sub-title section1"> Methods</h3><p>The training data collection used a retrospective set of prostate MR studies, from SPIE‐AAPM‐NCI PROSTATEx Challenges held in 2017. Three hundred twenty biopsy samples of lesions from 201 prostate cancer patients were diagnosed and identified as clinically significant (malignant) or not significant (benign). All studies included T2‐weighted (T2W), proton density‐weighted (PD‐W), dynamic contrast enhanced (DCE) and diffusion‐weighted (DW) imaging. After registration and lesion‐based normalization, a CNN with four convolutional layers were developed and trained on 10‐fold cross validation. The features from intermediate layers were then extracted as input to wELM to test the discriminative power of each individual layer. The best performing model from the 10 folds was chosen to be tested on the holdout cohort from two sources. Feature maps after each convolutional layer were then visualized to monitor the trend, as the layer propagated. Scatter plotting was used to visualize the transformation of data distribution. Finally, a class activation map was generated to highlight the region of interest based on the model perspective.</p></section><section class="article-section__content"><h3 class="article-section__sub-title section1"> Results</h3><p>Experimental trials indicated that the best input for CNN was a modality combination of T2W, apparent diffusion coefficient (ADC) and DWIb50. The convolutional features from CNN paired with a weighted extreme learning classifier showed substantial performance compared to a CNN end‐to‐end training model. The feature map visualization reveals similar findings on natural images where lower layers tend to learn lower level features such as edges, intensity changes, etc, while higher layers learn more abstract and task‐related concept such as the lesion region. The generated saliency map revealed that the model was able to focus on the region of interest where the lesion resided and filter out background information, including prostate boundary, rectum, etc.</p></section><section class="article-section__content"><h3 class="article-section__sub-title section1"> Conclusions</h3><p>This work designs a customized workflow for the small and imbalanced data set of prostate mpMRI where features were extracted from a deep learning model and then analyzed by a traditional machine learning classifier. In addition, this work contributes to revealing how deep learning models interpret mpMRI for prostate cancer patients stratification.</p></section>

Deep learning-based prediction of tumor aggressiveness in RCC using multiparametric MRI: a pilot study

Deep Learning for Accurate Diagnosis of Liver Tumor Based on Magnetic Resonance Imaging and Clinical Data

A deep learning model based on dynamic contrast-enhanced magnetic resonance imaging enables accurate prediction of benign and malignant breast lessons

Deep learning prediction of pathological complete response, residual cancer burden, and progression-free survival in breast cancer patients

Multitask deep learning for prediction of microvascular invasion and recurrence-free survival in hepatocellular carcinoma based on MRI images

A deep learning fusion model with evidence-based confidence level analysis for differentiation of malignant and benign breast tumors using dynamic contrast enhanced MRI

Predicting malignancy in breast lesions: enhancing accuracy with fine-tuned convolutional neural network models

Deep learning-based diagnosis and survival prediction of patients with renal cell carcinoma from primary whole slide images

Deep Learning to Distinguish Benign from Malignant Renal Lesions Based on Routine MR Imaging

Deep Learning-Based Multi-Model Prediction for Disease-Free Survival Status of Patients with Clear Cell Renal Cell Carcinoma after Surgery: a Multicenter Cohort Study.

Noninvasive assessment of breast cancer molecular subtypes on multiparametric MRI using convolutional neural network with transfer learning

Clinical T1/2 renal cell carcinoma: multiparametric dynamic contrast-enhanced MRI features-based model for the prediction of individual adverse pathology

Machine and Deep Learning Prediction Of Prostate Cancer Aggressiveness Using Multiparametric MRI

Study on the classification of benign and malignant breast lesions using a multi-sequence breast MRI fusion radiomics and deep learning model

A deep dive into understanding tumor foci classification using multiparametric MRI based on convolutional neural network

Deep learning for risk prediction in patients with nasopharyngeal carcinoma using multi-parametric MRIs

Multiparametric MRI-Based Deep Learning Models for Preoperative Prediction of Tumor Deposits in Rectal Cancer and Prognostic Outcome

Deep learning-based multimodel prediction for disease-free survival status of patients with clear cell renal cell carcinoma after surgery: a multicenter cohort study

Multimodal data integration for predicting progression risk in castration-resistant prostate cancer using deep learning: a multicenter retrospective study

Prior to Initiation of Chemotherapy, Can We Predict Breast Tumor Response? Deep Learning Convolutional Neural Networks Approach Using a Breast MRI Tumor Dataset

Deep Learning for Predicting Distant Metastasis in Patients with Nasopharyngeal Carcinoma Based on Pre-Radiotherapy Magnetic Resonance Imaging