STFNet: Enhanced and Lightweight Spatiotemporal Fusion Network for Wearable Human Activity Recognition
Hailin Zou,Zijie Chen,Chenyang Zhang,Anran Yuan,Binbin Wang,Lei Wang,Jianqing Li,Yuanyuan Pan
DOI: https://doi.org/10.1109/jsen.2024.3373444
IF: 4.3
2024-01-01
IEEE Sensors Journal
Abstract:Human activity recognition using sensor data has become a research hotspot in the field of ubiquitous computing and has a wide range of application scenarios in real life. Effectively extracting and fusing the rich spatiotemporal information contained in the sensor data from various sensors and multiple wearable devices is a challenging task. Moreover, for practical deployment applications, the limited computational and storage resources of edge devices also impose requirements on the model size and inference speed. Hence, we propose STFNet, an enhanced and lightweight spatiotemporal fusion network for wearable human activity recognition. In which, we innovatively divide the extraction and fusion process of spatiotemporal features into three stages, including sensing feature extraction, temporal feature extraction, and channel fusion, to learn incrementally and reduce information loss. We first extract shallow intra-channel features and learn local inter-channel information by using a lightweight local attention mechanism. Then, the learned features are fed into a hierarchical convolution structure with residual connections to extract multi-scale and comprehensive temporal features. Finally, we use graph convolutions to further fuse the deep inter-channel information. By replacing the conventional global attention mechanism with a local one and using hierarchical convolution to replace dilated or multi-kernel convolution, STFNet achieves the purpose of lightweight design. In the experiments, STFNet performs better on public datasets than the selected WHAR models published in recent years and demonstrates its lightweight characteristics on edge device deployment.
engineering, electrical & electronic,instruments & instrumentation,physics, applied