Multi-modal Fusion Deep Learning Model for Excavated Soil Heterogeneous Data with Efficient Classification
Qi-Meng Guo,Liang-Tong Zhan,Zhen-Yu Yin,Hang Feng,Guang-Qian Yang,Yun-Min Chen
DOI: https://doi.org/10.1016/j.compgeo.2024.106697
IF: 5.218
2024-01-01
Computers and Geotechnics
Abstract:Multimodal fusion, a cutting-edge method aiming to integrate heterogeneous data for prediction tasks, has received limited attention in geotechnical engineering. This study focuses on classifying excavated soils with large quantities rapidly and finely. By using an excavated soil information collecting system at the largest soil transferring platform in China, 3,243 groups of multi-source heterogenous data were created, including soil images, spatial series data of cone index curves, time series data of TDR waveforms, discrete net weight data, and textual descriptions of soil morphology. After data augmentation, a big dataset containing 23,122 sets with labels based on the Unified Soil Classification System, moisture content, and mineral composition was created. Multimodal deep learning models were trained using different fusion strategies: 7 early fusion, 3 intermediate fusion, and 2 late fusion cases. Performance metrics were evaluated, including loss, accuracy, F1 Score, precision, recall, specificity, NPV, FPR, and AUROC. Results showed that intermediate fusions performed best, while late fusions performed the poorest. The two-stage intermediate fusion with five modalities achieved the best results, achieving an accuracy above 0.99 on the test set. This multimodal fusion approach effectively explores the correlation between multi-source heterogeneous data of soils, leading to scientific and engineering value in geotechnics.