Abstract:Recent years have witnessed significant success of convolutional neural networks (CNNs) in human activity recognition (HAR) using wearable sensors. Nevertheless, prior works have an obvious drawback. An activity sample may contain heterogeneous sensor modalities from different body parts. Moreover, the significance of each modality will change over time. Because a normal convolution filter usually samples activity data at a fixed regular grid, it is hard to capture salient features of activities along different sensor modalities or time intervals. What is the best filter form for activity recognition still remains a challenging task. In this article, to resolve this issue, we present a new deformable convolutional network for recognizing human activities from intricate sensory data. Specifically, the learned offsets and the feature amplitudes are added into standard convolution, which can be modulated to allow more free-form deformation over the sampling grid for sensory data. Comparing previous results, we achieve state-of-the-art recognition accuracies, e.g., 82.91%, 80.02%, 97.35%, and 99.21%, respectively, on several benchmark HAR datasets, including OPPORTUNITY, UNIMIB-SHAR, USC-HAD, and WISDM, hence indicating the advantage of the proposed method. The visual analysis is provided, which shows that the deformation could be conditioned on different input activity samples. The receptive field and the sampling locations can be adjusted in an adaptive manner, which leads to better interpretability for deep model behaviors. Installing PyTorch on a Raspberry Pi 3 B plus system, we evaluate actual run time of the deformable model. The results show that the deformable filter is able to still maintain almost the same inference time, which is very beneficial for activity recognition tasks. Our work can promote further research by leveraging intermodulating information to connect the deformable co-volution and attention modules.

Beyond Grids: Scaling Up Continuous Kernels Via Adaptive Point Representations for Sensor-Based Human Activity Recognition

Deep Neural Networks for Sensor-Based Human Activity Recognition Using Selective Kernel Convolution

Cloud-Edge Collaborative Human Activity Recognition System Design and Implementation.

Multiscale Deep Feature Learning for Human Activity Recognition Using Wearable Sensors

Deep Convolutional Networks With Tunable Speed–Accuracy Tradeoff for Human Activity Recognition Using Wearables

Cross-Attention Enhanced Pyramid Multi-Scale Networks for Sensor-based Human Activity Recognition

Human activity recognition using wearable sensors by heterogeneous convolutional neural networks

Real-time Human Activity Recognition Using Conditionally Parametrized Convolutions on Mobile and Wearable Devices

Shallow Convolutional Neural Networks for Human Activity Recognition Using Wearable Sensors

Large Receptive Field Attention: An Innovation in Decomposing Large-Kernel Convolution for Sensor-Based Activity Recognition

Temporal-Spatial Dynamic Convolutional Neural Network for Human Activity Recognition Using Wearable Sensors

Human Activity Recognition with Convolutional Neural Netowrks

A Multidimensional Parallel Convolutional Connected Network Based on Multisource and Multimodal Sensor Data for Human Activity Recognition

A Multi-dimensional Parallel Convolutional Connected Network Based on Multi-source and Multi-modal Sensor Data for Human Activity Recognition

Continual learning in sensor-based human activity recognition: An empirical benchmark analysis

AS-APF: Encoding time series as images for human activity recognition with SK-based convolutional networks

HAR-GCNN: Deep Graph CNNs for Human Activity Recognition From Highly Unlabeled Mobile Sensor Data

Dual-Branch Interactive Networks on Multichannel Time Series for Human Activity Recognition

Deformable Convolutional Networks for Multimodal Human Activity Recognition Using Wearable Sensors

Innovative Dual-Decoupling CNN with Layer-wise Temporal-Spatial Attention for Sensor-Based Human Activity Recognition

Continuous Human Activity Recognition With Distributed Radar Sensor Networks and CNN–RNN Architectures