Abstract:The popularity of convolutional architecture has made sensor-based human activity recognition (HAR) become one primary beneficiary. By simply superimposing multiple convolution layers, the local features can be effectively captured from multi-channel time series sensor data, which could output high-performance activity prediction results. On the other hand, recent years have witnessed great success of Transformer model, which uses powerful self-attention mechanism to handle long-range sequence modeling tasks, hence avoiding the shortcoming of local feature representations caused by convolutional neural networks (CNNs). In this paper, we seek to combine the merits of CNN and Transformer to model multi-channel time series sensor data, which might provide compelling recognition performance with fewer parameters and FLOPs based on lightweight wearable devices. To this end, we propose a new Dual-branch Interactive Network (DIN) that inherits the advantages from both CNN and Transformer to handle multi-channel time series for HAR. Specifically, the proposed framework utilizes two-stream architecture to disentangle local and global features by performing conv-embedding and patch-embedding, where a co-attention mechanism is used to adaptively fuse global-to-local and local-to-global feature representations. We perform extensive experiments on three mainstream HAR benchmark datasets including PAMAP2, WISDM, and OPPORTUNITY, which verify that our method consistently outperforms several state-of-the-art baselines, reaching an F1-score of 92.05%, 98.17%, and 91.55% respectively with fewer parameters and FLOPs. In addition, the practical execution time is validated on an embedded Raspberry Pi P3 system, which demonstrates that our approach is adequately efficient for real-time HAR implementations and deserves as a better alternative in ubiquitous HAR computing scenario. Our model code will be released soon.

Predicting Complex Activities from Ongoing Multivariate Time Series

A Novel Real-Time Multi-Person Activity Recognition System Using Millimeter-Wave SISO Radar Data

Sample Intercorrelation-Based Multidomain Fusion Network for Aquatic Human Activity Recognition Using Millimeter-Wave Radar.

A Novel Multiperson Activity Recognition Algorithm Based on Point Clouds Measured by Millimeter-Wave MIMO Radar

KCAR: A knowledge-driven approach for concurrent activity recognition

Continuous Human Activity Recognition With Distributed Radar Sensor Networks and CNN–RNN Architectures

Complex Activity Recognition Using Time Series Pattern Dictionary Learned from Ubiquitous Sensors

Sensor-based Human Activity Recognition System with a Multilayered Model Using Time Series Shapelets

Inferring Ongoing Human Activities Based on Recurrent Self-Organizing Map Trajectory.

Simultaneous Implementation Features Extraction and Recognition Using C3D Network for WiFi-based Human Activity Recognition

Multi-channel Time Series Decomposition Network For Generalizable Sensor-Based Activity Recognition

Human Activity Recognition Based on Mixed CNN With Radar Multi-Spectrogram

Human Activity Recognition Based on Deep Learning and Micro-Doppler Radar Data

CAE-MAS: Convolutional Autoencoder Interference Cancellation for Multiperson Activity Sensing With FMCW Microwave Radar

Modeling Interdependent and Periodic Real-World Action Sequences

Continuous Human Activity Recognition using a MIMO Radar for Transitional Motion Analysis

Deep Convolutional Neural Network with RNNs for Complex Activity Recognition Using Wrist-Worn Wearable Sensor Data

Human Activity Recognition based on Dynamic Spatio-Temporal Relations

A Pattern Mining Approach to Sensor-Based Human Activity Recognition

Dual-Branch Interactive Networks on Multichannel Time Series for Human Activity Recognition

Deep Learning Techniques for Radar-Based Continuous Human Activity Recognition