Abstract:The popularity of convolutional architecture has made sensor-based human activity recognition (HAR) become one primary beneficiary. By simply superimposing multiple convolution layers, the local features can be effectively captured from multi-channel time series sensor data, which could output high-performance activity prediction results. On the other hand, recent years have witnessed great success of Transformer model, which uses powerful self-attention mechanism to handle long-range sequence modeling tasks, hence avoiding the shortcoming of local feature representations caused by convolutional neural networks (CNNs). In this paper, we seek to combine the merits of CNN and Transformer to model multi-channel time series sensor data, which might provide compelling recognition performance with fewer parameters and FLOPs based on lightweight wearable devices. To this end, we propose a new Dual-branch Interactive Network (DIN) that inherits the advantages from both CNN and Transformer to handle multi-channel time series for HAR. Specifically, the proposed framework utilizes two-stream architecture to disentangle local and global features by performing conv-embedding and patch-embedding, where a co-attention mechanism is used to adaptively fuse global-to-local and local-to-global feature representations. We perform extensive experiments on three mainstream HAR benchmark datasets including PAMAP2, WISDM, and OPPORTUNITY, which verify that our method consistently outperforms several state-of-the-art baselines, reaching an F1-score of 92.05%, 98.17%, and 91.55% respectively with fewer parameters and FLOPs. In addition, the practical execution time is validated on an embedded Raspberry Pi P3 system, which demonstrates that our approach is adequately efficient for real-time HAR implementations and deserves as a better alternative in ubiquitous HAR computing scenario. Our model code will be released soon.

DanHAR: Dual Attention Network for multimodal human activity recognition using wearable sensors

Wearable Sensor Based Multimodal Human Activity Recognition Exploiting the Diversity of Classifier Ensemble.

Multihead-Res-SE Residual Network with Attention for Human Activity Recognition

A Deep Dilated Convolutional Self-attention Model for Multimodal Human Activity Recognition

Multi-STMT: Multi-Level Network for Human Activity Recognition Based on Wearable Sensors

Innovative Dual-Decoupling CNN with Layer-wise Temporal-Spatial Attention for Sensor-Based Human Activity Recognition

A Novel Deep Multifeature Extraction Framework Based on Attention Mechanism Using Wearable Sensor Data for Human Activity Recognition

DiamondNet: A Neural-Network-Based Heterogeneous Sensor Attentive Fusion for Human Activity Recognition

Human Activity Recognition Using Attention-Mechanism-Based Deep Learning Feature Combination

Dual-Branch Interactive Networks on Multichannel Time Series for Human Activity Recognition

Triple Cross-Domain Attention on Human Activity Recognition Using Wearable Sensors

Understanding and Improving Channel Attention for Human Activity Recognition by Temporal-Aware and Modality-Aware Embedding

Privacy-preserving Activity Recognition Using Multimodal Sensors in Smart Office

Attention-based LSTM Network for Wearable Human Activity Recognition

A Multi-dimensional Parallel Convolutional Connected Network Based on Multi-source and Multi-modal Sensor Data for Human Activity Recognition

GRU with Dual Attentions for Sensor-Based Human Activity Recognition

Plug-and-play Multi-Dimensional Attention Module for Accurate Human Activity Recognition

A Multidimensional Parallel Convolutional Connected Network Based on Multisource and Multimodal Sensor Data for Human Activity Recognition

TCN-attention-HAR: human activity recognition based on attention mechanism time convolutional network

Deep Wavelet Convolutional Neural Networks for Multimodal Human Activity Recognition Using Wearable Inertial Sensors

Multi-level Contrast Network for Wearables-based Joint Activity Segmentation and Recognition.