Ischemic stroke as the first manifestation of hepatic epithelioid hemangioendothelioma.

P. Zis,A. Assi,Dimitrios Kravaritis,V. Sevastianos

DOI: https://doi.org/10.1016/j.jstrokecerebrovasdis.2013.09.030

2014-03-01

Abstract:

What problem does this paper attempt to address?

Using Multi-Task Learning to Improve Diagnostic Performance of Convolutional Neural Networks

Mengjie Fang,Di Dong,Ruijia Sun,Li Fan,Yingshi Sun,Shiyuan Liu,Jie Tian

DOI: https://doi.org/10.1117/12.2512153

2019-01-01

Abstract:Due to the complex biological and physical mechanisms, the correlations between the classification objects of clinical tasks and the medical imaging phenotype are always ambiguous and implied, which makes it difficult to train a powerful diagnostic convolutional neural network (CNN) model efficiently. In this study, we propose a generic multi-task learning (MTL) CNN framework to achieve higher classification accuracy and better generalization. The proposed framework is designed to carry out the major diagnostic task and several auxiliary tasks simultaneously. It encourages the models to learn more beneficial representation following the underlying relation among patients’ clinical characteristics, obvious imaging findings and quantitative imaging phenotype. We evaluate our approach on two clinical applications, namely advanced gastric cancer (AGC) serosa invasion diagnosis and discrimination of lung invasive adenocarcinoma manifesting as ground-glass nodule (GGN). Two datasets are utilized, which contain 357 AGC patients’ venous phase contrast-enhanced CT volumes and 236 GGN patients’ non-contrast CT volumes respectively. Several subjective CT morphology characteristics and common clinical characteristics are collected and used as the auxiliary tasks. To evaluate the generality of our strategy, CNNs with and without natural image-based pre-training are successively incorporated into the framework. The experimental results demonstrate that the proposed MTL CNN framework is able to improve the diagnostic performance significantly (7.4%-12.8% AUC increase and 3.5%-7.9% accuracy increase).
A Multi-modal Fusion Framework Based on Multi-task Correlation Learning for Cancer Prognosis Prediction

Kaiwen Tan,Weixian Huang,Xiaofeng Liu,Jinlong Hu,Shoubin Dong

DOI: https://doi.org/10.1016/j.artmed.2022.102260

2022-01-22

Abstract:Morphological attributes from histopathological images and molecular profiles from genomic data are important information to drive diagnosis, prognosis, and therapy of cancers. By integrating these heterogeneous but complementary data, many multi-modal methods are proposed to study the complex mechanisms of cancers, and most of them achieve comparable or better results from previous single-modal methods. However, these multi-modal methods are restricted to a single task (e.g., survival analysis or grade classification), and thus neglect the correlation between different tasks. In this study, we present a multi-modal fusion framework based on multi-task correlation learning (MultiCoFusion) for survival analysis and cancer grade classification, which combines the power of multiple modalities and multiple tasks. Specifically, a pre-trained ResNet-152 and a sparse graph convolutional network (SGCN) are used to learn the representations of histopathological images and mRNA expression data respectively. Then these representations are fused by a fully connected neural network (FCNN), which is also a multi-task shared network. Finally, the results of survival analysis and cancer grade classification output simultaneously. The framework is trained by an alternate scheme. We systematically evaluate our framework using glioma datasets from The Cancer Genome Atlas (TCGA). Results demonstrate that MultiCoFusion learns better representations than traditional feature extraction methods. With the help of multi-task alternating learning, even simple multi-modal concatenation can achieve better performance than other deep learning and traditional methods. Multi-task learning can improve the performance of multiple tasks not just one of them, and it is effective in both single-modal and multi-modal data.

Machine Learning,Artificial Intelligence,Computer Vision and Pattern Recognition
Multi-task fusion for improving mammography screening data classification

Maria Wimmer,Gert Sluiter,David Major,Dimitrios Lenis,Astrid Berg,Theresa Neubauer,Katja Bühler

DOI: https://doi.org/10.1109/TMI.2021.3129068

2021-12-01

Abstract:Machine learning and deep learning methods have become essential for computer-assisted prediction in medicine, with a growing number of applications also in the field of mammography. Typically these algorithms are trained for a specific task, e.g., the classification of lesions or the prediction of a mammogram's pathology status. To obtain a comprehensive view of a patient, models which were all trained for the same task(s) are subsequently ensembled or combined. In this work, we propose a pipeline approach, where we first train a set of individual, task-specific models and subsequently investigate the fusion thereof, which is in contrast to the standard model ensembling strategy. We fuse model predictions and high-level features from deep learning models with hybrid patient models to build stronger predictors on patient level. To this end, we propose a multi-branch deep learning model which efficiently fuses features across different tasks and mammograms to obtain a comprehensive patient-level prediction. We train and evaluate our full pipeline on public mammography data, i.e., DDSM and its curated version CBIS-DDSM, and report an AUC score of 0.962 for predicting the presence of any lesion and 0.791 for predicting the presence of malignant lesions on patient level. Overall, our fusion approaches improve AUC scores significantly by up to 0.04 compared to standard model ensembling. Moreover, by providing not only global patient-level predictions but also task-specific model results that are related to radiological features, our pipeline aims to closely support the reading workflow of radiologists.

Image and Video Processing,Computer Vision and Pattern Recognition
Multimodal microscopic imaging with deep learning for highly effective diagnosis of breast cancer

Jinjin Wu,Zhibing Xu,Linwei Shang,Zihan Wang,Suwei Zhou,Hui Shang,Huijie Wang,Jianhua Yin

DOI: https://doi.org/10.1016/j.optlaseng.2023.107667

IF: 5.666

2023-05-25

Optics and Lasers in Engineering

Abstract:Accurate and rapid diagnosis of breast cancer is particularly important since it becomes the primary malignant tumor threatening women's health. Although pathological diagnosis with H&E staining section is the gold standard in tumor diagnosis, it is hard to meet quick intraoperative inspection because of its complex procedures and hysteresis. This study adopts multimodal microscopic imaging technology (bright-field imaging, auto-fluorescence imaging and orthogonal polarization imaging) combined with deep learning to achieve rapid intelligent diagnosis of breast cancer by getting the rich information of tissue morphology, content and structure of collagen in tissue slices. By using both fusion classification models of multimodal images at pixel level and decision level, the AUC score and accuracy is 0.9366 and 89.01% for the pixel fusion classification, as well as 0.9421 and 87.53% for the decision fusion classification, respectively. The results suggest that the multimodal microscopic imaging technique proposed in this study has unique advantages in accuracy, speed and feasibility in clinic for breast cancer diagnosis, and has strong clinical potential and application prospects when combined with deep learning.

optics
Prior Clinico‐Radiological Features Informed Multi‐Modal MR Images Convolution Neural Network: A novel deep learning framework for prediction of lymphovascular invasion in breast cancer

Hong Zheng,Lian Jian,Li Li,Wen Liu,Wei Chen

DOI: https://doi.org/10.1002/cam4.6932

IF: 4.711

2024-01-19

Cancer Medicine

Abstract:Background Current methods utilizing preoperative magnetic resonance imaging (MRI)‐based radiomics for assessing lymphovascular invasion (LVI) in patients with early‐stage breast cancer lack precision, limiting the options for surgical planning. Purpose This study aimed to develop a sophisticated deep learning framework called "Prior Clinico‐Radiological Features Informed Multi‐Modal MR Images Convolutional Neural Network (PCMM‐Net)" to improve the accuracy of LVI prediction in breast cancer. By incorporating multiparameter MRI and prior clinical knowledge, PCMM‐Net should enhance the precision of LVI assessment. Methods A total of 341 patients with breast cancer were randomly divided into training and validation groups at a ratio of 7:3. Imaging features were extracted from T1‐weighted, T2‐weighted, and contrast‐enhanced T1‐weighted MRI sequences. Stepwise univariate and multivariate logistic regression were employed to establish a clinico‐radiological model for LVI prediction. The radiomics model was built using redundancy and the least absolute shrinkage and selection operator. Then, two deep learning frameworks were developed: the Multi‐Modal MR Images Convolutional Neural Network (MM‐Net), which does not consider prior radiological features, and PCMM‐Net, which incorporates multiparameter MRI and prior clinical knowledge. Receiver operating characteristic curves were used, and the corresponding areas under the curves (AUCs) were calculated for evaluation. Results PCMM‐Net achieved the highest AUC of 0.843. The clinico‐radiological features displayed the lowest AUC value of 0.743, followed by MM‐Net with an AUC of 0.774, and radiomics with an AUC of 0.795. Conclusions This study introduces PCMM‐Net, an innovative deep learning framework that integrates prior clinico‐radiological features for accurate LVI prediction in breast cancer. PCMM‐Net demonstrates excellent diagnostic performance and facilitates the application of precision medicine.

oncology
Multi-instance Multi-task Learning for Joint Clinical Outcome and Genomic Profile Predictions from the Histopathological Images

Wei Shao,Hang Shi,Jianxin Liu,Yingli Zuo,Liang Sun,TianSong Xia,Wanyuan Chen,Peng Wan,JianPeng Sheng,Qi Zhu,Daoqiang Zhang

DOI: https://doi.org/10.1109/tmi.2024.3362852

IF: 10.6

2024-01-01

IEEE Transactions on Medical Imaging

Abstract:With the remarkable success of digital histopathology and the deep learning technology, many whole-slide pathological images (WSIs) based deep learning models are designed to help pathologists diagnose human cancers. Recently, rather than predicting categorical variables as in cancer diagnosis, several deep learning studies are also proposed to estimate the continuous variables such as the patients' survival or their transcriptional profile. However, most of the existing studies focus on conducting these predicting tasks separately, which overlooks the useful intrinsic correlation among them that can boost the prediction performance of each individual task. In addition, it is sill challenge to design the WSI-based deep learning models, since a WSI is with huge size but annotated with coarse label. In this study, we propose a general multi-instance multi-task learning framework (HistMIMT) for multi-purpose prediction from WSIs. Specifically, we firstly propose a novel multi-instance learning module (TMICS) considering both common and specific task information across different tasks to generate bag representation for each individual task. Then, a soft-mask based fusion module with channel attention (SFCA) is developed to leverage useful information from the related tasks to help improve the prediction performance on target task. We evaluate our method on three cancer cohorts derived from the Cancer Genome Atlas (TCGA). For each cohort, our multi-purpose prediction tasks range from cancer diagnosis, survival prediction and estimating the transcriptional profile of gene TP53. The experimental results demonstrated that HistMIMT can yield better outcome on all clinical prediction tasks than its competitors.

engineering, biomedical,imaging science & photographic technology, electrical & electronic,computer science, interdisciplinary applications,radiology, nuclear medicine & medical imaging
Study on the classification of benign and malignant breast lesions using a multi-sequence breast MRI fusion radiomics and deep learning model

Wenjiang Wang,Jiaojiao Li,Zimeng Wang,Yanjun Liu,Fei Yang,Shujun Cui

DOI: https://doi.org/10.1016/j.ejro.2024.100607

2024-10-23

European Journal of Radiology Open

Abstract:Purpose To develop a multi-modal model combining multi-sequence breast MRI fusion radiomics and deep learning for the classification of benign and malignant breast lesions, to assist clinicians in better selecting treatment plans. Methods A total of 314 patients who underwent breast MRI examinations were included. They were randomly divided into training, validation, and test sets in a ratio of 7:1:2. Subsequently, features of T1-weighted images (T1WI), T2-weighted images (T2WI), and dynamic contrast-enhanced MRI (DCE-MRI) were extracted using the convolutional neural network ResNet50 for fusion, and then combined with radiomic features from the three sequences. The following models were established: T1 model, T2 model, DCE model, DCE_T1_T2 model, and DCE_T1_T2_rad model. The performance of the models was evaluated by the area under the receiver operating characteristic (ROC) curve (AUC), accuracy, sensitivity, specificity, positive predictive value, and negative predictive value. The differences between the DCE_T1_T2_rad model and the other four models were compared using the Delong test, with a P -value < 0.05 considered statistically significant. Results The five models established in this study performed well, with AUC values of 0.53 for the T1 model, 0.62 for the T2 model, 0.79 for the DCE model, 0.94 for the DCE_T1_T2 model, and 0.98 for the DCE_T1_T2_rad model. The DCE_T1_T2_rad model showed statistically significant differences ( P < 0.05) compared to the other four models. Conclusion The use of a multi-modal model combining multi-sequence breast MRI fusion radiomics and deep learning can effectively improve the diagnostic performance of breast lesion classification.
Multi-path synergic fusion deep neural network framework for breast mass classification using digital breast tomosynthesis

Linjing Wang,Chao Zheng,Wentao Chen,Qiang He,Xin Li,Shuxu Zhang,Genggeng Qin,Weiguo Chen,Jun Wei,Peiliang Xie,Linghong Zhou,Xuetao Wang,Xin Zhen

DOI: https://doi.org/10.1088/1361-6560/abaeb7

IF: 3.5

2020-12-01

Physics in Medicine and Biology

Abstract:Abstract Objective. To develop and evaluate a multi-path synergic fusion (MSF) deep neural network model for breast mass classification using digital breast tomosynthesis (DBT). Methods. We retrospectively collected 441 patients who had undergone DBT in which the regions of interest (ROIs) covering the malignant/benign breast mass were extracted for model training and validation. In the proposed MSF framework, three multifaceted representations of the breast mass (gross mass, overview, and mass background) are extracted from the ROIs and independently processed by a multi-scale multi-level features enforced DenseNet (MMFED). The three MMFED sub-models are finally fused at the decision level to generate the final prediction. The advantages of the MMFED over the original DenseNet, as well as different fusion strategies embedded in MSF, were comprehensively compared. Results. The MMFED was observed to be superior to the original DenseNet, and multiple channel fusions in the MSF outperformed the single-channel MMFED and double-channel fusion with the best classification scores of area under the receiver operating characteristic (ROC) curve (87.03%), Accuracy (81.29%), Sensitivity (74.57%), and Specificity (84.53%) via the weighted fusion method embedded in MSF. The decision level fusion-based MSF was significantly better (in terms of the ROC curve) than the feature concatenation-based fusion ( p < 0.05), the single MMFED using a fused three-channel image ( p < 0.04), and the multiple MMFED end-to-end training ( p < 0.004). Conclusions. Integrating multifaceted representations of the breast mass tends to increase benign/malignant mass classification performance and the proposed methodology was verified to be a promising tool to assist in clinical breast cancer screening.

engineering, biomedical,radiology, nuclear medicine & medical imaging
MRI‐Based Breast Cancer Classification and Localization by Multiparametric Feature Extraction and Combination Using Deep Learning

Chao Cong,Xiaoguang Li,Chunlai Zhang,Jing Zhang,Kaixiang Sun,Lianluyi Liu,Bharath Ambale‐Venkatesh,Xiao Chen,Yi Wang

DOI: https://doi.org/10.1002/jmri.28713

IF: 4.4

2023-04-05

Journal of Magnetic Resonance Imaging

Abstract:Background Deep learning (DL) have been reported feasible in breast MRI. However, the effectiveness of DL method in mpMRI combinations for breast cancer detection has not been well investigated. Purpose To implement a DL method for breast cancer classification and detection using feature extraction and combination from multiple sequences. Study Type Retrospective. Population A total of 569 local cases as internal cohort (50.2 ± 11.2 years; 100% female), divided among training (218), validation (73) and testing (278); 125 cases from a public dataset as the external cohort (53.6 ± 11.5 years; 100% female). Field Strength/Sequence T1‐weighted imaging and dynamic contrast‐enhanced MRI (DCE‐MRI) with gradient echo sequences, T2‐weighted imaging (T2WI) with spin‐echo sequences, diffusion‐weighted imaging with single‐shot echo‐planar sequence and at 1.5‐T. Assessment A convolutional neural network and long short‐term memory cascaded network was implemented for lesion classification with histopathology as the ground truth for malignant and benign categories and contralateral breasts as healthy category in internal/external cohorts. BI‐RADS categories were assessed by three independent radiologists as comparison, and class activation map was employed for lesion localization in internal cohort. The classification and localization performances were assessed with DCE‐MRI and non‐DCE sequences, respectively. Statistical Tests Sensitivity, specificity, area under the curve (AUC), DeLong test, and Cohen's kappa for lesion classification. Sensitivity and mean squared error for localization. A P‐value <0.05 was considered statistically significant. Results With the optimized mpMRI combinations, the lesion classification achieved an AUC = 0.98/0.91, sensitivity = 0.96/0.83 in the internal/external cohorts, respectively. Without DCE‐MRI, the DL‐based method was superior to radiologists' readings (AUC 0.96 vs. 0.90). The lesion localization achieved sensitivities of 0.97/0.93 with DCE‐MRI/T2WI alone, respectively. Data Conclusion The DL method achieved high accuracy for lesion detection in the internal/external cohorts. The classification performance with a contrast agent‐free combination is comparable to DCE‐MRI alone and the radiologists' reading in AUC and sensitivity. Evidence Level 3. Technical Efficacy Stage 2.

radiology, nuclear medicine & medical imaging
Identification of Luminal A breast cancer by using deep learning analysis based on multi-modal images

Menghan Liu,Shuai Zhang,Yanan Du,Xiaodong Zhang,Dawei Wang,Wanqing Ren,Jingxiang Sun,Shiwei Yang,Guang Zhang

DOI: https://doi.org/10.3389/fonc.2023.1243126

2023-11-17

Abstract:Purpose: To evaluate the diagnostic performance of a deep learning model based on multi-modal images in identifying molecular subtype of breast cancer. Materials and methods: A total of 158 breast cancer patients (170 lesions, median age, 50.8 ± 11.0 years), including 78 Luminal A subtype and 92 non-Luminal A subtype lesions, were retrospectively analyzed and divided into a training set (n = 100), test set (n = 45), and validation set (n = 25). Mammography (MG) and magnetic resonance imaging (MRI) images were used. Five single-mode models, i.e., MG, T2-weighted imaging (T2WI), diffusion weighting imaging (DWI), axial apparent dispersion coefficient (ADC), and dynamic contrast-enhanced MRI (DCE-MRI), were selected. The deep learning network ResNet50 was used as the basic feature extraction and classification network to construct the molecular subtype identification model. The receiver operating characteristic curve were used to evaluate the prediction efficiency of each model. Results: The accuracy, sensitivity and specificity of a multi-modal tool for identifying Luminal A subtype were 0.711, 0.889, and 0.593, respectively, and the area under the curve (AUC) was 0.802 (95% CI, 0.657- 0.906); the accuracy, sensitivity, and AUC were higher than those of any single-modal model, but the specificity was slightly lower than that of DCE-MRI model. The AUC value of MG, T2WI, DWI, ADC, and DCE-MRI model was 0.593 (95%CI, 0.436-0.737), 0.700 (95%CI, 0.545-0.827), 0.564 (95%CI, 0.408-0.711), 0.679 (95%CI, 0.523-0.810), and 0.553 (95%CI, 0.398-0.702), respectively. Conclusion: The combination of deep learning and multi-modal imaging is of great significance for diagnosing breast cancer subtypes and selecting personalized treatment plans for doctors.
A Deep Learning Model for Predicting Molecular Subtype of Breast Cancer by Fusing Multiple Sequences of DCE-MRI From Two Institutes

Xiaoyang Xie,Haowen Zhou,Mingze Ma,Ji Nie,Weibo Gao,Jinman Zhong,Xin Cao,Xiaowei He,Jinye Peng,Yuqing Hou,Fengjun Zhao,Xin Chen

DOI: https://doi.org/10.1016/j.acra.2024.03.002

Abstract:Rationale and objectives: To evaluate the performance of deep learning (DL) in predicting different breast cancer molecular subtypes using DCE-MRI from two institutes. Materials and methods: This retrospective study included 366 breast cancer patients from two institutes, divided into training (n = 292), validation (n = 49) and testing (n = 25) sets. We first transformed the public DCE-MRI appearance to ours to alleviate small-data-size and class-imbalance issues. Second, we developed a multi-branch convolutional-neural-network (MBCNN) to perform molecular subtype prediction. Third, we assessed the MBCNN with different regions of interest (ROIs) and fusion strategies, and compared it to previous DL models. Area under the curve (AUC) and accuracy (ACC) were used to assess different models. Delong-test was used for the comparison of different groups. Results: MBCNN achieved the optimal performance under intermediate fusion and ROI size of 80 pixels with appearance transformation. It outperformed CNN and convolutional long-short-term-memory (CLSTM) in predicting luminal B, HER2-enriched and TN subtypes, but without demonstrating statistical significance except against CNN in TN subtypes, with testing AUCs of 0.8182 vs. [0.7208, 0.7922] (p=0.44, 0.80), 0.8500 vs. [0.7300, 0.8200] (p=0.36, 0.70) and 0.8900 vs. [0.7600, 0.8300] (p=0.03, 0.63), respectively. When predicting luminal A, MBCNN outperformed CNN with AUCs of 0.8571 vs. 0.7619 (p=0.08) without achieving statistical significance, and is comparable to CLSTM. For four-subtype prediction, MBCNN achieved an ACC of 0.64, better than CNN and CLSTM models with ACCs of 0.48 and 0.52, respectively. Conclusion: Developed DL model with the feature extraction and fusion of DCE-MRI from two institutes enabled preoperative prediction of breast cancer molecular subtypes with high diagnostic performance.
Deep learning radiomics based on multimodal imaging for distinguishing benign and malignant breast tumours

Guoxiu Lu,Ronghui Tian,Wei Yang,Ruibo Liu,Dongmei Liu,Zijie Xiang,Guoxu Zhang

DOI: https://doi.org/10.3389/fmed.2024.1402967

IF: 3.9

2024-07-05

Frontiers in Medicine

Abstract:Objectives This study aimed to develop a deep learning radiomic model using multimodal imaging to differentiate benign and malignant breast tumours. Methods Multimodality imaging data, including ultrasonography (US), mammography (MG), and magnetic resonance imaging (MRI), from 322 patients (112 with benign breast tumours and 210 with malignant breast tumours) with histopathologically confirmed breast tumours were retrospectively collected between December 2018 and May 2023. Based on multimodal imaging, the experiment was divided into three parts: traditional radiomics, deep learning radiomics, and feature fusion. We tested the performance of seven classifiers, namely, SVM, KNN, random forest, extra trees, XGBoost, LightGBM, and LR, on different feature models. Through feature fusion using ensemble and stacking strategies, we obtained the optimal classification model for benign and malignant breast tumours. Results In terms of traditional radiomics, the ensemble fusion strategy achieved the highest accuracy, AUC, and specificity, with values of 0.892, 0.942 [0.886–0.996], and 0.956 [0.873–1.000], respectively. The early fusion strategy with US, MG, and MRI achieved the highest sensitivity of 0.952 [0.887–1.000]. In terms of deep learning radiomics, the stacking fusion strategy achieved the highest accuracy, AUC, and sensitivity, with values of 0.937, 0.947 [0.887–1.000], and 1.000 [0.999–1.000], respectively. The early fusion strategies of US+MRI and US+MG achieved the highest specificity of 0.954 [0.867–1.000]. In terms of feature fusion, the ensemble and stacking approaches of the late fusion strategy achieved the highest accuracy of 0.968. In addition, stacking achieved the highest AUC and specificity, which were 0.997 [0.990–1.000] and 1.000 [0.999–1.000], respectively. The traditional radiomic and depth features of US+MG + MR achieved the highest sensitivity of 1.000 [0.999–1.000] under the early fusion strategy. Conclusion This study demonstrated the potential of integrating deep learning and radiomic features with multimodal images. As a single modality, MRI based on radiomic features achieved greater accuracy than US or MG. The US and MG models achieved higher accuracy with transfer learning than the single-mode or radiomic models. The traditional radiomic and depth features of US+MG + MR achieved the highest sensitivity under the early fusion strategy, showed higher diagnostic performance, and provided more valuable information for differentiation between benign and malignant breast tumours.

medicine, general & internal
Application of a Convolutional Neural Network for Multitask Learning to Simultaneously Predict Microvascular Invasion and Vessels that Encapsulate Tumor Clusters in Hepatocellular Carcinoma

Tongjia Chu,Chen Zhao,Jian Zhang,Kehang Duan,Mingyang Li,Tianqi Zhang,Shengnan Lv,Huan Liu,Feng Wei

DOI: https://doi.org/10.1245/s10434-022-12000-6

IF: 4.339

2022-06-26

Annals of Surgical Oncology

Abstract:Abstract Background Hepatocellular carcinoma (HCC) is the fourth most common cause of cancer death worldwide, and the prognosis remains dismal. In this study, two pivotal factors, microvascular invasion (MVI) and vessels encapsulating tumor clusters (VETC) were preoperatively predicted simultaneously to assess prognosis. Methods A total of 133 HCC patients who underwent surgical resection and preoperative gadolinium ethoxybenzyl-diethylenetriaminepentaacetic acid (Gd-EOB-DTPA)-enhanced magnetic resonance imaging (MRI) were included. The statuses of MVI and VETC were obtained from the pathological report and CD34 immunohistochemistry, respectively. A three-dimensional convolutional neural network (3D CNN) for single-task learning aimed at MVI prediction and for multitask learning aimed at simultaneous prediction of MVI and VETC was established by using multiphase Gd-EOB-DTPA-enhanced MRI. Results The 3D CNN for single-task learning achieved an area under receiver operating characteristics curve (AUC) of 0.896 (95% CI: 0.797–0.994). Multitask learning with simultaneous extraction of MVI and VETC features improved the performance of MVI prediction, with an AUC value of 0.917 (95% CI: 0.825–1.000), and achieved an AUC value of 0.860 (95% CI: 0.728–0.993) for the VETC prediction. The multitask learning framework could stratify high- and low-risk groups regarding overall survival ( p < 0.0001) and recurrence-free survival ( p < 0.0001), revealing that patients with MVI+/VETC+ were associated with poor prognosis. Conclusions A deep learning framework based on 3D CNN for multitask learning to predict MVI and VETC simultaneously could improve the performance of MVI prediction while assessing the VETC status. This combined prediction can stratify prognosis and enable individualized prognostication in HCC patients before curative resection.

oncology,surgery
A deep dive into understanding tumor foci classification using multiparametric MRI based on convolutional neural network

Weiwei Zong,Joon K. Lee,Chang Liu,Eric N. Carver,Aharon M. Feldman,Branislava Janic,Mohamed A. Elshaikh,Milan V. Pantelic,David Hearshen,Indrin J. Chetty,Benjamin Movsas,Ning Wen

DOI: https://doi.org/10.1002/mp.14255

IF: 4.506

2020-06-12

Medical Physics

Abstract:<section class="article-section__content"><h3 class="article-section__sub-title section1"> Purpose</h3><p>Deep learning models have had a great success in disease classifications using large data pools of skin cancer images or lung X‐rays. However, data scarcity has been the roadblock of applying deep learning models directly on prostate multiparametric MRI (mpMRI). Although model interpretation has been heavily studied for natural images for the past few years, there has been a lack of interpretation of deep learning models trained on medical images.</p><p>In this paper, an efficient convolutional neural network (CNN) was developed and the model interpretation at various convolutional layers was systematically analyzed to improve the understanding of how CNN interprets multimodality medical images and the predictive powers of features at each layer. The problem of small sample size was addressed by feeding the intermediate features into a traditional classification algorithm known as weighted extreme learning machine (wELM), with imbalanced distribution among output categories taken into consideration.</p></section><section class="article-section__content"><h3 class="article-section__sub-title section1"> Methods</h3><p>The training data collection used a retrospective set of prostate MR studies, from SPIE‐AAPM‐NCI PROSTATEx Challenges held in 2017. Three hundred twenty biopsy samples of lesions from 201 prostate cancer patients were diagnosed and identified as clinically significant (malignant) or not significant (benign). All studies included T2‐weighted (T2W), proton density‐weighted (PD‐W), dynamic contrast enhanced (DCE) and diffusion‐weighted (DW) imaging. After registration and lesion‐based normalization, a CNN with four convolutional layers were developed and trained on 10‐fold cross validation. The features from intermediate layers were then extracted as input to wELM to test the discriminative power of each individual layer. The best performing model from the 10 folds was chosen to be tested on the holdout cohort from two sources. Feature maps after each convolutional layer were then visualized to monitor the trend, as the layer propagated. Scatter plotting was used to visualize the transformation of data distribution. Finally, a class activation map was generated to highlight the region of interest based on the model perspective.</p></section><section class="article-section__content"><h3 class="article-section__sub-title section1"> Results</h3><p>Experimental trials indicated that the best input for CNN was a modality combination of T2W, apparent diffusion coefficient (ADC) and DWIb50. The convolutional features from CNN paired with a weighted extreme learning classifier showed substantial performance compared to a CNN end‐to‐end training model. The feature map visualization reveals similar findings on natural images where lower layers tend to learn lower level features such as edges, intensity changes, etc, while higher layers learn more abstract and task‐related concept such as the lesion region. The generated saliency map revealed that the model was able to focus on the region of interest where the lesion resided and filter out background information, including prostate boundary, rectum, etc.</p></section><section class="article-section__content"><h3 class="article-section__sub-title section1"> Conclusions</h3><p>This work designs a customized workflow for the small and imbalanced data set of prostate mpMRI where features were extracted from a deep learning model and then analyzed by a traditional machine learning classifier. In addition, this work contributes to revealing how deep learning models interpret mpMRI for prostate cancer patients stratification.</p></section>

radiology, nuclear medicine & medical imaging
Multi-task Learning-Based Histologic Subtype Classification of Non-Small Cell Lung Cancer.

Kun Chen,Manning Wang,Zhijian Song

DOI: https://doi.org/10.1007/s11547-023-01621-w

2023-01-01

La radiologia medica

Abstract:Purpose In clinical applications, accurate histologic subtype classification of lung cancer is important for determining appropriate treatment plans. The purpose of this paper is to evaluate the role of multi-task learning in the classification of adenocarcinoma and squamous cell carcinoma. Material and methods In this paper, we propose a novel multi-task learning model for histologic subtype classification of non-small cell lung cancer based on computed tomography (CT) images. The model consists of a histologic subtype classification branch and a staging branch, which share a part of the feature extraction layers and are simultaneously trained. By optimizing on the two tasks simultaneously, our model could achieve high accuracy in histologic subtype classification of non-small cell lung cancer without relying on physician’s precise labeling of tumor areas. In this study, 402 cases from The Cancer Imaging Archive (TCIA) were used in total, and they were split into training set ( n = 258), internal test set ( n = 66) and external test set ( n = 78). Results Compared with the radiomics method and single-task networks, our multi-task model could reach an AUC of 0.843 and 0.732 on internal and external test set, respectively. In addition, multi-task network can achieve higher accuracy and specificity than single-task network. Conclusion Compared with the radiomics methods and single-task networks, our multi-task learning model could improve the accuracy of histologic subtype classification of non-small cell lung cancer by sharing network layers, which no longer relies on the physician’s precise labeling of lesion regions and could further reduce the manual workload of physicians.
Breast Tumor Classification based on MRI-US Images by Disentangling Modality Features

Mengyun Qiao,Chengcheng Liu,Zeju Li,Jin Zhou,Qin Xiao,Shichong Zhou,Cai Chang,Gu Yajia,Yi Guo,Yuanyuan Wang

DOI: https://doi.org/10.1109/jbhi.2022.3140236

IF: 7.7

2022-01-01

IEEE Journal of Biomedical and Health Informatics

Abstract:Dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) and ultrasound (US), which are two common modalities for clinical breast tumor diagnosis besides Mammograms, can provide different and complementary information for the same tumor regions. Although many machine learning methods have been proposed for breast tumor classification based on either single modality, it remains unclear how to further boost the classification performance by utilizing paired multi-modality information with different dimensions. In this paper, we propose MRI-US multi-modality network (MUM-Net) to classify breast tumor into different subtypes based on 3D MR and 2D US images. The key insight of MUM-Net is that we explicitly distill modality-agnostic features for tumor classification. Specifically, we first adopt a discrimination-adaption module to decompose features into modality-agnostic and modality-specific ones with min-max training strategies. Then, we propose a feature fusion module to increase the compactness of the modality-agnostic features by utilizing an affinity matrix with nearest neighbour selection. We build a paired MRI-US breast tumor classification dataset containing 502 cases with three clinical indicators to validate the proposed method. In three tasks including lymph node metastasis, histological grade and Ki-67 level, MUM-Net achieves AUC scores of 0.8581, 0.8965 and 0.8577, outperforming other counterparts which are based on single task or single modality by a wide margin. In addition, we find that the extracted modality-agnostic features can help the network focus on the tumor regions in both modalities.

computer science, interdisciplinary applications,mathematical & computational biology,medical informatics, information systems
Study on the differential diagnosis of benign and malignant breast lesions using a deep learning model based on multimodal images

Yanan Du,Dawei Wang,Menghan Liu,Xiaodong Zhang,Wanqing Ren,Jingxiang Sun,Chao Yin,Shiwei Yang,Li Zhang

DOI: https://doi.org/10.4103/jcrt.jcrt_1796_23

2024-04-01

Journal of Cancer Research and Therapeutics

Abstract:ABSTRACT Objective: To establish a multimodal model for distinguishing benign and malignant breast lesions. Materials and Methods: Clinical data, mammography, and MRI images (including T2WI, diffusion-weighted images (DWI), apparent diffusion coefficient (ADC), and DCE-MRI images) of 132 benign and breast cancer patients were analyzed retrospectively. The region of interest (ROI) in each image was marked and segmented using MATLAB software. The mammography, T2WI, DWI, ADC, and DCE-MRI models based on the ResNet34 network were trained. Using an integrated learning method, the five models were used as a basic model, and voting methods were used to construct a multimodal model. The dataset was divided into a training set and a prediction set. The accuracy, sensitivity, specificity, positive predictive value, and negative predictive value of the model were calculated. The diagnostic efficacy of each model was analyzed using a receiver operating characteristic curve (ROC) and an area under the curve (AUC). The diagnostic value was determined by the DeLong test with statistically significant differences set at P < 0.05. Results: We evaluated the ability of the model to classify benign and malignant tumors using the test set. The AUC values of the multimodal model, mammography model, T2WI model, DWI model, ADC model and DCE-MRI model were 0.943, 0.645, 0.595, 0.905, 0.900, and 0.865, respectively. The diagnostic ability of the multimodal model was significantly higher compared with that of the mammography and T2WI models. However, compared with the DWI, ADC, and DCE-MRI models, there was no significant difference in the diagnostic ability of these models. Conclusion: Our deep learning model based on multimodal image training has practical value for the diagnosis of benign and malignant breast lesions.

oncology
Breast MRI Segmentation and Ki-67 High- and Low-Expression Prediction Algorithm Based on Deep Learning

Yuan-Zhe Li,Yin-Hui Huang,Xian-yan Su,Zhen-qi Gu,Qing-Quan Lai,Jing Huang,Shu-Ting Li,Yi Wang

DOI: https://doi.org/10.1155/2022/1770531

IF: 2.809

2022-01-01

Computational and Mathematical Methods in Medicine

Abstract:Background and Objective. Breast cancer is a common malignant tumor that seriously threatens the health of women in my country and even around the world. The proliferation marker Ki-67 has been utilized to distinguish luminal B from luminal A tumors and is a reliable indicator of more aggressive breast cancer growth. If a reliable prediction method for breast cancer patients to avoid invasive damage can be found to predict Ki-67 before pathological examination, it will be very beneficial for doctors to formulate later treatment plans and provide more useful treatment options. Methodology. This paper proposes a tumor segmentation and prediction framework based on the combination of improved attention U-Net and SVM. The framework first improves on attention U-Net by introducing coefficients for learning multidimensional attention. Make the attention mechanism more aware of the main situation in the segmentation process. At the same time, the segmented breast MRI results and corresponding labels were input into the SVM classifier to accurately predict the expression of Ki-67. Results. The DSC, PPV, and sensitivity of our combined model are 0.94, 0.93, and 0.94, respectively, with better segmentation performance. And we compare with the segmentation frameworks of other papers and find that our combined model can make accurate segmentation of breast tumors. Conclusion. Our method can adapt to the variability of breast tumors and segment breast tumors accurately and efficiently. In the future, it can be widely used in clinical practice, so as to help the clinic better formulate a reasonable diagnosis and treatment plan for breast cancer patients.
A Novel Deep Learning Model for Breast Tumor Ultrasound Image Classification with Lesion Region Perception

Jinzhu Wei,Haoyang Zhang,Jiang Xie

DOI: https://doi.org/10.3390/curroncol31090374

2024-08-30

Current Oncology

Abstract:Multi-task learning (MTL) methods are widely applied in breast imaging for lesion area perception and classification to assist in breast cancer diagnosis and personalized treatment. A typical paradigm of MTL is the shared-backbone network architecture, which can lead to information sharing conflicts and result in the decline or even failure of the main task's performance. Therefore, extracting richer lesion features and alleviating information-sharing conflicts has become a significant challenge for breast cancer classification. This study proposes a novel Multi-Feature Fusion Multi-Task (MFFMT) model to effectively address this issue. Firstly, in order to better capture the local and global feature relationships of lesion areas, a Contextual Lesion Enhancement Perception (CLEP) module is designed, which integrates channel attention mechanisms with detailed spatial positional information to extract more comprehensive lesion feature information. Secondly, a novel Multi-Feature Fusion (MFF) module is presented. The MFF module effectively extracts differential features that distinguish between lesion-specific characteristics and the semantic features used for tumor classification, and enhances the common feature information of them as well. Experimental results on two public breast ultrasound imaging datasets validate the effectiveness of our proposed method. Additionally, a comprehensive study on the impact of various factors on the model's performance is conducted to gain a deeper understanding of the working mechanism of the proposed framework.

oncology
A Multi-Task Convolutional Neural Network for Lesion Region Segmentation and Classification of Non-Small Cell Lung Carcinoma

Zhao Wang,Yuxin Xu,Linbo Tian,Qingjin Chi,Fengrong Zhao,Rongqi Xu,Guilei Jin,Yansong Liu,Junhui Zhen,Sasa Zhang

DOI: https://doi.org/10.3390/diagnostics12081849

IF: 3.6

2022-08-01

Diagnostics

Abstract:Targeted therapy is an effective treatment for non-small cell lung cancer. Before treatment, pathologists need to confirm tumor morphology and type, which is time-consuming and highly repetitive. In this study, we propose a multi-task deep learning model based on a convolutional neural network for joint cancer lesion region segmentation and histological subtype classification, using magnified pathological tissue images. Firstly, we constructed a shared feature extraction channel to extract abstract information of visual space for joint segmentation and classification learning. Then, the weighted losses of segmentation and classification tasks were tuned to balance the computing bias of the multi-task model. We evaluated our model on a private in-house dataset of pathological tissue images collected from Qilu Hospital of Shandong University. The proposed approach achieved Dice similarity coefficients of 93.5% and 89.0% for segmenting squamous cell carcinoma (SCC) and adenocarcinoma (AD) specimens, respectively. In addition, the proposed method achieved an accuracy of 97.8% in classifying SCC vs. normal tissue and an accuracy of 100% in classifying AD vs. normal tissue. The experimental results demonstrated that our method outperforms other state-of-the-art methods and shows promising performance for both lesion region segmentation and subtype classification.

medicine, general & internal

Ischemic stroke as the first manifestation of hepatic epithelioid hemangioendothelioma.

Using Multi-Task Learning to Improve Diagnostic Performance of Convolutional Neural Networks

A Multi-modal Fusion Framework Based on Multi-task Correlation Learning for Cancer Prognosis Prediction

Multi-task fusion for improving mammography screening data classification

Multimodal microscopic imaging with deep learning for highly effective diagnosis of breast cancer

Prior Clinico‐Radiological Features Informed Multi‐Modal MR Images Convolution Neural Network: A novel deep learning framework for prediction of lymphovascular invasion in breast cancer

Multi-instance Multi-task Learning for Joint Clinical Outcome and Genomic Profile Predictions from the Histopathological Images

Study on the classification of benign and malignant breast lesions using a multi-sequence breast MRI fusion radiomics and deep learning model

Multi-path synergic fusion deep neural network framework for breast mass classification using digital breast tomosynthesis

MRI‐Based Breast Cancer Classification and Localization by Multiparametric Feature Extraction and Combination Using Deep Learning

Identification of Luminal A breast cancer by using deep learning analysis based on multi-modal images

A Deep Learning Model for Predicting Molecular Subtype of Breast Cancer by Fusing Multiple Sequences of DCE-MRI From Two Institutes

Deep learning radiomics based on multimodal imaging for distinguishing benign and malignant breast tumours

Application of a Convolutional Neural Network for Multitask Learning to Simultaneously Predict Microvascular Invasion and Vessels that Encapsulate Tumor Clusters in Hepatocellular Carcinoma

A deep dive into understanding tumor foci classification using multiparametric MRI based on convolutional neural network

Multi-task Learning-Based Histologic Subtype Classification of Non-Small Cell Lung Cancer.

Breast Tumor Classification based on MRI-US Images by Disentangling Modality Features

Study on the differential diagnosis of benign and malignant breast lesions using a deep learning model based on multimodal images

Breast MRI Segmentation and Ki-67 High- and Low-Expression Prediction Algorithm Based on Deep Learning

A Novel Deep Learning Model for Breast Tumor Ultrasound Image Classification with Lesion Region Perception

A Multi-Task Convolutional Neural Network for Lesion Region Segmentation and Classification of Non-Small Cell Lung Carcinoma