Abstract:Federated human activity recognition (FHAR) has attracted much attention due to its great potential in privacy protection. Existing FHAR methods can collaboratively learn a global activity recognition model based on unimodal or multimodal data distributed on different local clients. However, it is still questionable whether existing methods can work well in a more common scenario where local data are from different modalities, e.g., some local clients may provide motion signals while others can only provide visual data. In this paper, we study a new problem of cross-modal federated human activity recognition (CM-FHAR), which is conducive to promote the large-scale use of the HAR model on more local devices. CM-FHAR has at least three dedicated challenges: (1) distributive common cross-modal feature learning, (2) modality-dependent discriminate feature learning, (3) modality imbalance issue. To address these challenges, we propose a modality-collaborative activity recognition network (MCARN), which can comprehensively learn a global activity classifier shared across all clients and multiple modality-dependent private activity classifiers. To produce modality-agnostic and modality-specific features, we learn an altruistic encoder and an egocentric encoder under the constraint of a separation loss and an adversarial modality discriminator collaboratively learned in hyper-sphere. To address the modality imbalance issue, we propose an angular margin adjustment scheme to improve the modality discriminator on modality-imbalanced data by enhancing the intra-modality compactness of the dominant modality and increase the inter-modality discrepancy. Moreover, we propose a relation-aware global-local calibration mechanism to constrain class-level pairwise relationships for the parameters of the private classifier. Finally, through decentralized optimization with alternative steps of adversarial local updating and modality-aware global aggregation, the proposed MCARN obtains state-of-the-art performance on both modality-balanced and modality-imbalanced data.

ATFA: Adversarial Time–Frequency Attention network for sensor-based multimodal human activity recognition

Augmented Adversarial Learning for Human Activity Recognition with Partial Sensor Sets

A Novel Deep Multifeature Extraction Framework Based on Attention Mechanism Using Wearable Sensor Data for Human Activity Recognition

Attention-based LSTM Network for Wearable Human Activity Recognition

B2C-AFM: Bi-Directional Co-Temporal and Cross-Spatial Attention Fusion Model for Human Action Recognition.

Attention-Based Deep Learning Framework for Human Activity Recognition with User Adaptation

MMTSA: Multimodal Temporal Segment Attention Network for Efficient Human Activity Recognition

DanHAR: Dual Attention Network for multimodal human activity recognition using wearable sensors

Privacy-preserving Activity Recognition Using Multimodal Sensors in Smart Office

Human Activity Recognition Method Based on FMCW Radar Sensor with Multi-Domain Feature Attention Fusion Network

Wearable Sensor-Based Residual Multifeature Fusion Shrinkage Networks for Human Activity Recognition

Channel Attention for Sensor-based Activity Recognition: Embedding Features into All Frequencies in DCT Domain

Attention-Based Sensor Fusion for Human Activity Recognition Using IMU Signals

Multi-Sensor Data Fusion and CNN-LSTM Model for Human Activity Recognition System

Multi-Stage Based Feature Fusion of Multi-Modal Data for Human Activity Recognition

Innovative Dual-Decoupling CNN with Layer-wise Temporal-Spatial Attention for Sensor-Based Human Activity Recognition

Beyond Isolated Frames: Enhancing Sensor-Based Human Activity Recognition through Intra- and Inter-Frame Attention

Multi-STMT: Multi-Level Network for Human Activity Recognition Based on Wearable Sensors

HAR-Net:Fusing Deep Representation and Hand-crafted Features for Human Activity Recognition

Cross-Modal Federated Human Activity Recognition

A Multi-dimensional Parallel Convolutional Connected Network Based on Multi-source and Multi-modal Sensor Data for Human Activity Recognition