Multi-Sensor Data Fusion Method Based on Self-Attention Mechanism

Xuezhu Lin,Shihan Chao,Dongming Yan,Lili Guo,Yue Liu,Lijuan Li

DOI: https://doi.org/10.3390/app132111992

2023-11-03

Applied Sciences

Abstract:In 3D reconstruction tasks, single-sensor data fusion based on deep learning is limited by the integrity and accuracy of the data, which reduces the accuracy and reliability of the fusion results. To address this issue, this study proposes a multi-sensor data fusion method based on a self-attention mechanism. A multi-sensor data fusion model for acquiring multi-source and multi-modal data is constructed, with the core component being a convolutional neural network with self-attention (CNN-SA), which employs CNNs to process multi-source and multi-modal data by extracting their features. Additionally, it introduces an SA mechanism to weigh and sum the features of different modalities, adaptively focusing on the importance of different modal data. This enables mutual support, complementarity, and correction among the multi-modal data. Experimental results demonstrate that the accuracy of the CNN-SA network is improved by 72.6%, surpassing the improvements of 29.9% for CNN-CBAM, 23.6% for CNN, and 11.4% for CNN-LSTM, exhibiting enhanced generalization capability, accuracy, and robustness. The proposed approach will contribute to the effectiveness of multi-sensor data fusion processing.

materials science, multidisciplinary,engineering,chemistry,physics, applied

What problem does this paper attempt to address?

The paper attempts to address the issue in 3D reconstruction tasks where the accuracy and reliability of fusion results are low due to the insufficiency of data completeness and accuracy in single sensor data fusion methods. To overcome this problem, the paper proposes a multi-sensor data fusion method based on the self-attention mechanism.

Specifically, the paper proposes a multi-sensor data fusion model that utilizes Convolutional Neural Networks (CNN) and the Self-Attention Mechanism (SA). This model can handle multi-source, multi-modal data and dynamically adjust the importance of different modal data through the self-attention mechanism, thereby achieving complementarity and support between multi-modal data, and improving the accuracy and robustness of data fusion.

The main contributions of the paper include:
1. **Multi-source, multi-modal data fusion**: A multi-sensor data fusion model is constructed, capable of acquiring and processing multi-source, multi-modal data from different sensors.
2. **Self-attention mechanism**: The self-attention mechanism is introduced to dynamically adjust the importance of different modal data, enhancing the model's ability to utilize different modal data.
3. **Experimental validation**: The effectiveness of the proposed method is validated through experiments, and the results show that the method outperforms existing methods in terms of accuracy, generalization ability, and robustness.

In summary, this paper aims to address the limitations of single sensor data fusion methods in 3D reconstruction tasks by proposing a multi-sensor data fusion method based on the self-attention mechanism, thereby improving the accuracy and reliability of data fusion.