Systematic Comparison of Deep-Learning Based Fusion Strategies for Multi-Modal Ultrasound in Diagnosis of Liver Cancer
Ming-De Li,Wei Li,Man-Xia Lin,Xin-Xin Lin,Hang-Tong Hu,Ying-Chen Wang,Si-Min Ruan,Ze-Rong Huang,Rui-Fang Lu,Lv Li,Ming Kuang,Ming-De Lu,Li-Da Chen,Wei Wang,Qing-hua Huang
DOI: https://doi.org/10.1016/j.neucom.2024.128257
IF: 6
2024-01-01
Neurocomputing
Abstract:For the diagnosis of liver cancer, conventional brightness mode (B-mode) can only provide morphological information. Multi-modal ultrasound, including shear-wave elastography (SWE) and contrast enhanced ultrasound (CEUS), can provide comprehensive diagnostic information on tumor microenvironment and tissue perfusion. The challenge is to effectively explore the multi-modal features of ultrasound. Besides, there are many fusion strategies currently available, but there is a lack of systematic comparative research on the various fusion strategies. In this study, we designed 'Lesions Pairing' to construct the dataset, addressing the challenge of small sample sizes in multi-modal learning. We then compared the effectiveness of different strategies and proposed hybrid-fusion strategies based on the combination of conventional layer-level fusion (i.e. early-fusion, mid-fusion and late-fusion), which can efficiently extract intra-/inter- modal information. Specifically, we first systematically compared different deep-learning-based fusion strategies for multi-modal ultrasound in the diagnosis of liver cancer. Secondly, based on the comparison results of a multimodal framework that integrates B-mode, SWE, CEUS ultrasound data, and clinical data simultaneously, we propose a hybrid-fusion strategies for the diagnosis of hepatocellular carcinoma and intrahepatic cholangiocarcinoma. The experimental results showed that the area under the curve of the early-late fusion strategy combined with clinical data was 0.9854, which was superior to other single mode and other fusion strategies, increasing by 13.8-25.88% and 2.22%-9.79%, respectively.