Abstract:Objective: To go beyond the deficiencies of the three conventional multimodal fusion strategies (i.e., input-, feature- and output-level fusion), we propose a bidirectional attention-aware fluid pyramid feature integrated fusion network (BAF-Net) with cross-modal interactions for multimodal medical image diagnosis and prognosis. Approach: BAF-Net is composed of two identical branches to preserve the unimodal features and one bidirectional attention-aware distillation stream to progressively assimilate cross-modal complements and to learn supplementary features in both bottom-up and top-down processes. Fluid pyramid connections were adopted to integrate the hierarchical features at different levels of the network, and channel-wise attention modules were exploited to mitigate cross-modal cross-level incompatibility. Furthermore, depth-wise separable convolution was introduced to fuse the cross-modal cross-level features to alleviate the increase in parameters to a great extent. The generalization abilities of BAF-Net were evaluated in terms of two clinical tasks: (1) An in-house PET-CT dataset with 174 patients for differentiation between lung cancer and pulmonary tuberculosis. (2) A public multicenter PET-CT head and neck cancer dataset with 800 patients from nine centers for overall survival prediction. Main results: On the LC-PTB dataset, improved performance was found in BAF-Net (AUC = 0.7342) compared with input-level fusion model (AUC = 0.6825; p < 0.05), feature-level fusion model (AUC = 0.6968; p = 0.0547), output-level fusion model (AUC = 0.7011; p < 0.05). On the H&N cancer dataset, BAF-Net (C-index = 0.7241) outperformed the input-, feature-, and output-level fusion model, with 2.95%, 3.77%, and 1.52% increments of C-index (p = 0.3336, 0.0479 and 0.2911, respectively). The ablation experiments demonstrated the effectiveness of all the designed modules regarding all the evaluated metrics in both datasets. Significance: Extensive experiments on two datasets demonstrated better performance and robustness of BAF-Net than three conventional fusion strategies and PET or CT unimodal network in terms of diagnosis and prognosis.

Multi-sensor fusion for body sensor network in medical human–robot interaction scenario

Multisensor Data Fusion for Robot Parts Recognition

Interoperability technology of sports health monitoring equipment based on multi-sensor information fusion

Multimode Data Fusion Based Remote Healthcare Framework

Multimodal data fusion framework enhanced robot-assisted minimally invasive surgery

Information Fusion of Multi-Sensor Images

A Novel Multi-Sensor Hybrid Fusion Framework

Multi-Modal Data Fusion in Enhancing Human-Machine Interaction for Robotic Applications: A Survey

Multi-modal bioelectrical signal fusion analysis based on different acquisition devices and scene settings: Overview, challenges, and novel orientation

A Review on Multisensor Data Fusion for Wearable Health Monitoring

Multi-channel Sensor Network Construction, Data Fusion and Challenges for Smart Home

Multi-Sensor Fusion Based Robot Self-Activity Recognition

Dezert-Smarandache Theory-Based Fusion for Human Activity Recognition in Body Sensor Networks

BAF-Net: bidirectional attention-aware fluid pyramid feature integrated multimodal fusion network for diagnosis and prognosis

Progress and Prospects of Multimodal Fusion Methods in Physical Human–Robot Interaction: A Review

A body sensor data fusion and deep recurrent neural network-based behavior recognition approach for robust healthcare

AdaptiveFusion: Adaptive Multi-Modal Multi-View Fusion for 3D Human Body Reconstruction

Automated Fusion of Multimodal Electronic Health Records for Better Medical Predictions

Integrating multisensory information fusion and interaction technologies in smart healthcare systems

Multi-sensor fusion federated learning method of human posture recognition for dual-arm nursing robots

A comprehensive survey on multimodal medical signals fusion for smart healthcare systems