Abstract:Gesture recognition has drawn considerable attention from many researchers owing to its wide range of applications. Although significant progress has been made in this field, previous works always focus on how to distinguish between different gesture classes, ignoring the influence of inner-class divergence caused by gesture-irrelevant factors. Meanwhile, for multimodal gesture recognition, feature or score fusion in the final stage is a general choice to combine the information of different modalities. Consequently, the gesture-relevant features in different modalities may be redundant, whereas the complementarity of modalities is not exploited sufficiently. To handle these problems, we propose a hierarchical gesture prototype framework to highlight gesture-relevant features such as poses and motions in this article. This framework consists of a sample-level prototype and a modal-level prototype. The sample-level gesture prototype is established with the structure of a memory bank, which avoids the distraction of gesture-irrelevant factors in each sample, such as the illumination, background, and the performers' appearances. Then the modal-level prototype is obtained via a generative adversarial network (GAN)-based subnetwork, in which the modal-invariant features are extracted and pulled together. Meanwhile, the modal-specific attribute features are used to synthesize the feature of other modalities, and the circulation of modality information helps to leverage their complementarity. Extensive experiments on three widely used gesture datasets demonstrate that our method is effective to highlight gesture-relevant features and can outperform the state-of-the-art methods.

Multi-modal learning for gesture recognition

Dynamic hand gesture recognition using hidden Markov models

Multimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM

Multi-Task and Multi-Modal Learning for RGB Dynamic Gesture Recognition

Modality-convolutions: Multi-modal gesture recognition based on convolutional neural network

Multi-modal Gesture Recognition with Voting-Based Dynamic Time Warping

Multimodal Gesture Recognition Based On Choquet Integral

Multi-Scale Attention 3D Convolutional Network for Multimodal Gesture Recognition

Seeking a Hierarchical Prototype for Multimodal Gesture Recognition

Multi-modal fusion for robust hand gesture recognition based on heterogeneous networks

Research on Unified Recognition Model and Algorithm for Multi-Modal Gestures

Multimodal Gesture Recognition Based on the ResC3D Network.

Semg-Based Gesture Recognition with Embedded Virtual Hand Poses and Adversarial Learning.

Surface Electromyography-based Gesture Recognition by Multi-view Deep Learning.

Multimodal Gesture Recognition for Mascot Robot System Based on Choquet Integral Using Camera and 3D Accelerometers Fusion

MVHANet: Multi-view Hierarchical Aggregation Network for Skeleton-Based Hand Gesture Recognition

Multi-layered Gesture Recognition with Kinect

Multi-Modality Gesture Detection And Recognition With Un-Supervision, Randomization And Discrimination

Multimode Gesture Recognition Algorithm Based on Convolutional Long Short-Term Memory Network

Multi Gesture Recognition: A Tracking Learning Detection Approach.

Gesture Recognition Based on Multi‐modal Feature Weight