The diffusion coefficient of a swollen microgel particle.

A. Routh,W. Zimmerman

DOI: https://doi.org/10.1016/S0021-9797(03)00124-3

IF: 9.9

2003-05-15

Journal of Colloid and Interface Science

Abstract:

What problem does this paper attempt to address?

RGC: Reliable Gesture Classification Via Wearables Using GANs-Based Data Augmentation.

Han Zhou,Ji Zhao,Yi Gao,Wei Dong

DOI: https://doi.org/10.1109/icpads53394.2021.00114

2021-01-01

Abstract:Gesture-related human-computer interaction systems have been developed with different purposes. Wearable-based gesture recognition is well studied and considered to be effective. However, unexpected body movement of a user, such as walking and turning, is one issue that affects the robustness of classification. Collecting a sufficient dataset for every unexpected movement would be time-consuming and labor-intensive. To reduce the burden on users and address the lack of training data, we proposed RGC, which is a framework for IMU data augmentation. RGC adopts Generative-Adversarial-Networks-based and rotation-based augmentation to enhance the diversity of the dataset, which helps the classifier learn more effective representations for gesture recognition. We collected a dataset of 21120 arm gesture samples under different kinds of unexpected movements to evaluate RGC. The experiments showed that our method provides an 8.8% to 1.1% accuracy improvement for different SOTA classifiers with different original dataset sizes.
Multi-Scale Attention 3D Convolutional Network for Multimodal Gesture Recognition

Huizhou Chen,Yunan Li,Huijuan Fang,Wentian Xin,Zixiang Lu,Qiguang Miao

DOI: https://doi.org/10.3390/s22062405

IF: 3.9

2022-03-21

Sensors

Abstract:Gesture recognition is an important direction in computer vision research. Information from the hands is crucial in this task. However, current methods consistently achieve attention on hand regions based on estimated keypoints, which will significantly increase both time and complexity, and may lose position information of the hand due to wrong keypoint estimations. Moreover, for dynamic gesture recognition, it is not enough to consider only the attention in the spatial dimension. This paper proposes a multi-scale attention 3D convolutional network for gesture recognition, with a fusion of multimodal data. The proposed network achieves attention mechanisms both locally and globally. The local attention leverages the hand information extracted by the hand detector to focus on the hand region, and reduces the interference of gesture-irrelevant factors. Global attention is achieved in both the human-posture context and the channel context through a dual spatiotemporal attention module. Furthermore, to make full use of the differences between different modalities of data, we designed a multimodal fusion scheme to fuse the features of RGB and depth data. The proposed method is evaluated using the Chalearn LAP Isolated Gesture Dataset and the Briareo Dataset. Experiments on these two datasets prove the effectiveness of our network and show it outperforms many state-of-the-art methods.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
Regional Attention with Architecture-Rebuilt 3D Network for RGB-D Gesture Recognition

Benjia Zhou,Yunan Li,Jun Wan

DOI: https://doi.org/10.48550/arXiv.2102.05348

2021-03-10

Abstract:Human gesture recognition has drawn much attention in the area of computer vision. However, the performance of gesture recognition is always influenced by some gesture-irrelevant factors like the background and the clothes of performers. Therefore, focusing on the regions of hand/arm is important to the gesture recognition. Meanwhile, a more adaptive architecture-searched network structure can also perform better than the block-fixed ones like Resnet since it increases the diversity of features in different stages of the network better. In this paper, we propose a regional attention with architecture-rebuilt 3D network (RAAR3DNet) for gesture recognition. We replace the fixed Inception modules with the automatically rebuilt structure through the network via Neural Architecture Search (NAS), owing to the different shape and representation ability of features in the early, middle, and late stage of the network. It enables the network to capture different levels of feature representations at different layers more adaptively. Meanwhile, we also design a stackable regional attention module called dynamic-static Attention (DSA), which derives a Gaussian guidance heatmap and dynamic motion map to highlight the hand/arm regions and the motion information in the spatial and temporal domains, respectively. Extensive experiments on two recent large-scale RGB-D gesture datasets validate the effectiveness of the proposed method and show it outperforms state-of-the-art methods. The codes of our method are available at: <a class="link-external link-https" href="https://github.com/zhoubenjia/RAAR3DNet" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
Multi-Task and Multi-Modal Learning for RGB Dynamic Gesture Recognition

Dinghao Fan,Hengjie Lu,Shugong Xu,Shan Cao

DOI: https://doi.org/10.1109/JSEN.2021.3123443

2021-10-29

Abstract:Gesture recognition is getting more and more popular due to various application possibilities in human-machine interaction. Existing multi-modal gesture recognition systems take multi-modal data as input to improve accuracy, but such methods require more modality sensors, which will greatly limit their application scenarios. Therefore we propose an end-to-end multi-task learning framework in training 2D convolutional neural networks. The framework can use the depth modality to improve accuracy during training and save costs by using only RGB modality during inference. Our framework is trained to learn a representation for multi-task learning: gesture segmentation and gesture recognition. Depth modality contains the prior information for the location of the gesture. Therefore it can be used as the supervision for gesture segmentation. A plug-and-play module named Multi-Scale-Decoder is designed to realize gesture segmentation, which contains two sub-decoder. It is used in the lower stage and higher stage respectively, and can help the network pay attention to key target areas, ignore irrelevant information, and extract more discriminant features. Additionally, the MSD module and depth modality are only used in the training stage to improve gesture recognition performance. Only RGB modality and network without MSD are required during inference. Experimental results on three public gesture recognition datasets show that our proposed method provides superior performance compared with existing gesture recognition frameworks. Moreover, using the proposed plug-and-play MSD in other 2D CNN-based frameworks also get an excellent accuracy improvement.

Computer Vision and Pattern Recognition
Short-Term Temporal Convolutional Networks for Dynamic Hand Gesture Recognition

Yi Zhang,Chong Wang,Ye Zheng,Jieyu Zhao,Yuqi Li,Xijiong Xie

DOI: https://doi.org/10.48550/arXiv.2001.05833

2020-01-01

Abstract:The purpose of gesture recognition is to recognize meaningful movements of human bodies, and gesture recognition is an important issue in computer vision. In this paper, we present a multimodal gesture recognition method based on 3D densely convolutional networks (3D-DenseNets) and improved temporal convolutional networks (TCNs). The key idea of our approach is to find a compact and effective representation of spatial and temporal features, which orderly and separately divide task of gesture video analysis into two parts: spatial analysis and temporal analysis. In spatial analysis, we adopt 3D-DenseNets to learn short-term spatio-temporal features effectively. Subsequently, in temporal analysis, we use TCNs to extract temporal features and employ improved Squeeze-and-Excitation Networks (SENets) to strengthen the representational power of temporal features from each TCNs' layers. The method has been evaluated on the VIVA and the NVIDIA Gesture Dynamic Hand Gesture Datasets. Our approach obtains very competitive performance on VIVA benchmarks with the classification accuracies of 91.54%, and achieve state-of-the art performance with 86.37% accuracy on NVIDIA benchmark.

Computer Vision and Pattern Recognition,Image and Video Processing
Dynamic Gesture Recognition Method Based on Improved R(2+1)D

Yupeng Huo,Jie Shen,Sheng Zhang,Li Wang

DOI: https://doi.org/10.1109/cvidl58838.2023.10166120

2023-01-01

Abstract:Currently, methods based on 3D convolutional neural networks have made significant progress in the field of dynamic gesture recognition. Dynamic gestures are highly redundant in both the temporal and spatial dimensions, and the complex environment during the recognition process can easily affect the final recognition results. Therefore, it is crucial to make the model focus on the important moments and regions of gesture movements and extract relevant salient spatiotemporal features to further improve model performance. To address this issue, this paper proposes a lightweight Temporal-Spatial-Channel attention (TSCA) module based on the R(2+1)D network. The module consists of two sub-modules: a Temporal-Channel attention (TCA) module and a Temporal-Spatial attention (TSA) module, with the goal of enabling the model to focus on important information along the spatial, channel, and temporal dimensions during gesture movements. Finally, the TSCA attention module is integrated into the R(2+1)D network, resulting in only a 2.8M increase in parameters, and achieves good performance on the IPN-Hand and NvGesture datasets.
Gesture recognition based on deep deformable 3D convolutional neural networks

Yifan Zhang,Lei Shi,Yi Wu,Ke Cheng,Jian Cheng,Hanqing Lu

DOI: https://doi.org/10.1016/j.patcog.2020.107416

IF: 8

2020-11-01

Pattern Recognition

Abstract:<p>Dynamic gesture recognition, which plays an essential role in human-computer interaction, has been widely investigated but not yet fully addressed. The challenge mainly lies in three folders: 1) to model both of the spatial appearance and the temporal evolution simultaneously; 2) to address the interference from the varied and complex background; 3) the requirement of real-time processing. In this paper, we address the above challenges by proposing a novel deep deformable 3D convolutional neural network for end-to-end learning, which not only gains impressive accuracy in challenging datasets but also can meet the requirement of the real-time processing. We propose three types of very deep 3D CNNs for gesture recognition, which can directly model the spatiotemporal information with their inherent hierarchical structure. To eliminate the background interference, a light-weight spatiotemporal deformable convolutional module is specially designed to augment the spatiotemporal sampling locations of the 3D convolution by learning additional offsets according to the preceding feature map. It can not only diversify the shape of the convolution kernel to better fit the appearance of the hands and arms, but also help the models pay more attention to the discriminative frames in the video sequence. The proposed method is evaluated on three challenging datasets, EgoGesture, Jester and Chalearn-IsoGD, and achieves the state-of-the-art performance on all of them. Our model ranked first on Jester's official leader-board until the submission time. The code and the trained models are released for better communication and future works<a class="workspace-trigger" href="#fn0001"><sup>1</sup></a>.</p>

computer science, artificial intelligence,engineering, electrical & electronic
Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition

Zitong Yu,Benjia Zhou,Jun Wan,Pichao Wang,Haoyu Chen,Xin Liu,Stan Z. Li,Guoying Zhao

DOI: https://doi.org/10.1109/tip.2021.3087348

IF: 10.6

2021-01-01

IEEE Transactions on Image Processing

Abstract:Gesture recognition has attracted considerable attention owing to its great potential in applications. Although the great progress has been made recently in multi-modal learning methods, existing methods still lack effective integration to fully explore synergies among spatio-temporal modalities effectively for gesture recognition. The problems are partially due to the fact that the existing manually designed network architectures have low efficiency in the joint learning of multi-modalities. In this paper, we propose the first neural architecture search (NAS)-based method for RGB-D gesture recognition. The proposed method includes two key components: 1) enhanced temporal representation via the proposed 3D Central Difference Convolution (3D-CDC) family, which is able to capture rich temporal context via aggregating temporal difference information; and 2) optimized backbones for multi-sampling-rate branches and lateral connections among varied modalities. The resultant multi-modal multi-rate network provides a new perspective to understand the relationship between RGB and depth modalities and their temporal dynamics. Comprehensive experiments are performed on three benchmark datasets (IsoGD, NvGesture, and EgoGesture), demonstrating the state-of-the-art performance in both single- and multi-modality settings. The code is available at https://github.com/ZitongYu/3DCDC-NAS.

computer science, artificial intelligence,engineering, electrical & electronic
Gesture Recognition with a 3-D Accelerometer

Jiahui Wu,Gang Pan,Daqing Zhang,Guande Qi,Shijian Li

DOI: https://doi.org/10.1007/978-3-642-02830-4_4

2009-01-01

Abstract:Gesture-based interaction, as a natural way for human-computer interaction, has a wide range of applications in ubiquitous computing environment. This paper presents an acceleration-based gesture recognition approach, called FDSVM ( Frame-based Descriptor and multi-class SVM), which needs only a wearable 3-dimensional accelerometer. With FDSVM, firstly, the acceleration data of a gesture is collected and represented by a frame-based descriptor, to extract the discriminative information. Then a SVM-based multi-class gesture classifier is built for recognition in the nonlinear gesture feature space. Extensive experimental results on a data set with 3360 gesture samples of 12 gestures over weeks demonstrate that the proposed FDSVM approach significantly outperforms other four methods: DTW, Naïve Bayes, C4.5 and HMM. In the user-dependent case, FDSVM achieves the recognition rate of 99.38% for the 4 direction gestures and 95.21% for all the 12 gestures. In the user-independent case, it obtains the recognition rate of 98.93% for 4 gestures and 89.29% for 12 gestures. Compared to other accelerometer-based gesture recognition approaches reported in literature FDSVM gives the best resulrs for both user-dependent and user-independent cases.
Dynamic Gesture Recognition Algorithm Based on 3D Convolutional Neural Network

Yuting Liu,Du Jiang,Haojie Duan,Ying Sun,Gongfa Li,Bo Tao,Juntong Yun,Ying Liu,Baojia Chen

DOI: https://doi.org/10.1155/2021/4828102

IF: 3.12

2021-08-16

Computational Intelligence and Neuroscience

Abstract:Gesture recognition is one of the important ways of human-computer interaction, which is mainly detected by visual technology. The temporal and spatial features are extracted by convolution of the video containing gesture. However, compared with the convolution calculation of a single image, multiframe image of dynamic gestures has more computation, more complex feature extraction, and more network parameters, which affects the recognition efficiency and real-time performance of the model. To solve above problems, a dynamic gesture recognition model based on CBAM-C3D is proposed. Key frame extraction technology, multimodal joint training, and network optimization with BN layer are used for making the network performance better. The experiments show that the recognition accuracy of the proposed 3D convolutional neural network combined with attention mechanism reaches 72.4% on EgoGesture dataset, which is improved greatly compared with the current main dynamic gesture recognition methods, and the effectiveness of the proposed algorithm is verified.

mathematical & computational biology,neurosciences
Dynamic Gesture Recognition Based on Two-Scale 3-D-ConvNeXt

Sida Hao,Min Fu,Xuefeng Liu,Bing Zheng

DOI: https://doi.org/10.1109/jsen.2023.3324479

IF: 4.3

2023-12-01

IEEE Sensors Journal

Abstract:As a straightforward method for human–machine interaction, gesture recognition is vital in many practical applications. However, effectively extracting spatiotemporal information from video is still a fundamental problem and designing an accurate and efficient network is a feasible solution. The ConvNeXt, renowned for its superior still image processing capabilities, is chosen as the basis of this work. Then, the network is extended to a 3-D pattern for dynamic data and a two-scale convolution kernel is introduced to focus on the hand region. Therefore, a novel two-scale 3-D-ConvNeXt network (TS3C-Net) is established. Furthermore, the Mixup, Cutmix data augmentation, and label smoothing regularization are also applied to enhance the performance further. The experiments show that the accuracy of the proposed TS3C-Net achieves 95.36%, 97.1%, and 87.55% on EgoGesture, Jester, and NVGesture datasets, respectively.

engineering, electrical & electronic,instruments & instrumentation,physics, applied
Dynamic Hand Gesture Recognition Based on 3D Hand Pose Estimation for Human-Robot Interaction

Qing Gao,Yongquan Chen,Zhaojie Ju,Yi Liang

DOI: https://doi.org/10.1109/jsen.2021.3059685

IF: 4.3

2021-01-01

IEEE Sensors Journal

Abstract:Dynamic hand gesture recognition is a challenging problem in the area of hand-based human–robot interaction (HRI), such as issues of a complex environment and dynamic perception. In the context of this problem, we learn from the principle of the data-glove-based hand gesture recognition method and propose a dynamic hand gesture recognition method based on 3D hand pose estimation. This method uses 3D hand pose estimation, data fusion and deep neural network to improve the recognition accuracy of dynamic hand gestures. First, a 2D hand pose estimation method based on OpenPose is improved to obtain a fast 3D hand pose estimation method. Second, the weighted sum fusion method is utilized to combine the RGB, depth and 3D skeleton data of hand gestures. Finally, a 3DCNN + ConvLSTM framework is used to identify and classify the combined dynamic hand gesture data. In the experiment, the proposed method is verified on a developed dynamic hand gesture database for HRI and gets 92.4% accuracy. Comparative experiment results verify the reliability and efficiency of the proposed method.

engineering, electrical & electronic,instruments & instrumentation,physics, applied
AL-MobileNet: a Novel Model for 2D Gesture Recognition in Intelligent Cockpit Based on Multi-Modal Data

Bin Wang,Liwen Yu,Bo Zhang

DOI: https://doi.org/10.1007/s10462-024-10930-z

IF: 9.588

2024-01-01

Artificial Intelligence Review

Abstract:As the degree of automotive intelligence increases, gesture recognition is gaining more attention in human-vehicle interaction. However, existing gesture recognition methods are computationally intensive and perform poorly in multi-modal sensor scenarios. This paper proposes a novel network structure, AL-MobileNet (MobileNet with Attention and Lightweight Modules), which can quickly and accurately estimate 2D gestures in RGB and infrared (IR) images. The innovations of this paper are as follows: Firstly, to enhance multi-modal data, we created a synthetic IR dataset based on real 2D gestures and employed a coarse-to-fine training approach. Secondly, to speed up the model's computation on edge devices, we introduced a new lightweight computational module called the Split Channel Attention Block (SCAB). Thirdly, to ensure the model maintains accuracy in large datasets, we incorporated auxiliary networks and Angle-Weighted Loss (AWL) into the backbone network. Experiments show that AL-MobileNet requires only 0.4 GFLOPs of computational power and 1.2 million parameters. This makes it 1.5 times faster than MobileNet and allows for quick execution on edge devices. AL-MobileNet achieved a running speed of up to 28 FPS on the Ambarella CV28. On both general datasets and our dataset, our algorithm achieved an average PCK0.2 score of 0.95. This indicates that the algorithm can quickly generate accurate 2D gestures. The demonstration of the algorithm can be reviewed in gesturebaolong.
Large-scale Isolated Gesture Recognition Using Convolutional Neural Networks

Pichao Wang,Wanqing Li,Song Liu,Zhimin Gao,Chang Tang,Philip Ogunbona

DOI: https://doi.org/10.48550/arXiv.1701.01814

2017-01-07

Abstract:This paper proposes three simple, compact yet effective representations of depth sequences, referred to respectively as Dynamic Depth Images (DDI), Dynamic Depth Normal Images (DDNI) and Dynamic Depth Motion Normal Images (DDMNI). These dynamic images are constructed from a sequence of depth maps using bidirectional rank pooling to effectively capture the spatial-temporal information. Such image-based representations enable us to fine-tune the existing ConvNets models trained on image data for classification of depth sequences, without introducing large parameters to learn. Upon the proposed representations, a convolutional Neural networks (ConvNets) based method is developed for gesture recognition and evaluated on the Large-scale Isolated Gesture Recognition at the ChaLearn Looking at People (LAP) challenge 2016. The method achieved 55.57\% classification accuracy and ranked $2^{nd}$ place in this challenge but was very close to the best performance even though we only used depth data.

Computer Vision and Pattern Recognition
GesID: 3D Gesture Authentication Based on Depth Camera and One-Class Classification

Xuan Wang,Jiro Tanaka

DOI: https://doi.org/10.3390/s18103265

2018-09-28

Abstract:Biometric authentication is popular in authentication systems, and gesture as a carrier of behavior characteristics has the advantages of being difficult to imitate and containing abundant information. This research aims to use three-dimensional (3D) depth information of gesture movement to perform authentication with less user effort. We propose an approach based on depth cameras, which satisfies three requirements: Can authenticate from a single, customized gesture; achieves high accuracy without an excessive number of gestures for training; and continues learning the gesture during use of the system. To satisfy these requirements respectively: We use a sparse autoencoder to memorize the single gesture; we employ data augmentation technology to solve the problem of insufficient data; and we use incremental learning technology for allowing the system to memorize the gesture incrementally over time. An experiment has been performed on different gestures in different user situations that demonstrates the accuracy of one-class classification (OCC), and proves the effectiveness and reliability of the approach. Gesture authentication based on 3D depth cameras could be achieved with reduced user effort.
Necessity for including quality of life diagnostics and therapy in routine oncological care: Evaluation by methods of clinical guideline implementation.

M. Klinkhammer-Schalke,M. Koller,C. Ehret,B. Steinger,B. Ernst,F. Hofstädter,W. Lorenz

DOI: https://doi.org/10.1200/jco.2006.24.18_suppl.18594

IF: 45.3

2006-06-20

Journal of Clinical Oncology

Abstract:18594 Background: Application of quality of life assessment has become an important topic in oncology. Current approaches, however, have focused on technical issues within single center settings overlooking the requirements of routine care. We report on implementing of structural elements of a quality of life diagnostics and therapy system (QoL system). It was designed as integral part of the health care program for breast cancer patients in a regional tumor center. METHODS The QoL system was implemented by a systematic approach (Med. Care, 39, Suppl.2, 2(2001). Its elements comprised (a) coordinating practitioners (CPs) who functioned as gate keepers for QoL related therapies, (b) experts in the QoL unit who provided QoL reports and specific therapeutic recommendations based on patients' responses to the EORTC questionnaire, (c) professional health care providers in psychotherapy, pain therapy, social support, physiotherapy/lymph drainage, and fitness (nutrition, sports), (d) clinicians (gynecologists) for patient recruitment and (e) opinion leaders for educational influence on the CPs to keep the QoL system in daily practice. Success of implementation (as indicated by CPs' QoL-related changes in knowledge, attitude and behavior) was evaluated in a prospective study with breast cancer patients. RESULTS The QoL system was applicable in all of 170 patients recruited between Dec. 2002-June 2004 (age 34-86yrs, UICC 0-4, breast conserving surgery 67%). From 75 CPs (mostly gynecologists) in the area only 39 fulfilled criteria for the QoL system, 1 refused to participate. 38 CPs showed knowledge about QoL following educational outreach visits. Concerning attitudes and behavior, 64% of CPs found the experts' reports comprehensible, 56% followed their recommendations, 20% made additional actions. Follow-up 4 months after first QoL measurement revealed improvements in pain, body image and social life (p < .05). CONCLUSION The study-in combining doctor's approach and patient's demands-showed success using methods of guideline implementation by improving QoL deficits within a routine oncological care. Effectiveness will be analyzed in an ongoing randomized trial. No significant financial relationships to disclose.
FGDSNet: A Lightweight Hand Gesture Recognition Network for Human Robot Interaction

Guoyu Zhou,Zhenchao Cui,Jing Qi

DOI: https://doi.org/10.1109/lra.2024.3362144

IF: 5.2

2024-04-01

IEEE Robotics and Automation Letters

Abstract:Computer vision-based gesture recognition methods play a significant role in robot visual gesture interaction. since of low accuracy leading by insuffcient feature representation and fusion, the existing gesture segmentation and recognition methods fail to meet the requirements of practical applications. To address these issues, a lightweight two-stage end-to-end gesture recognition network called Fusing Gate Dual Stages Network (FGDSNet) is proposed. This network adopts a dual-branch network structure in the segmentation stage. Existing dual-branch network models often directly fuse detailed features and semantic features, which leads to detailed information being obscured by blurry semantic information. Additionally, there are redundant issues in the feature maps at different levels during the network inference process. Therefore, we embed Cosine Similarity-KL Divergence Attention Module (CoSKLAM) and Gate Filtering Module (GFM) between the local detail branch and the contextual semantic branch. The role of these two modules is to facilitate the fusion of local and global features during the feature extraction process and filter out redundant information. Finally, the segmentation result and original gesture image are used as inputs for the recognition network to predict gesture categories. The relevant experiments show that the proposed network performs well in both gesture segmentation and gesture recognition, while also having real-time inference speed and a smaller parameter size.

robotics
A Deep Learning Framework for Recognizing both Static and Dynamic Gestures

Osama Mazhar,Sofiane Ramdani,Andrea Cherubini

DOI: https://doi.org/10.48550/arXiv.2006.06321

2021-03-17

Abstract:Intuitive user interfaces are indispensable to interact with the human centric smart environments. In this paper, we propose a unified framework that recognizes both static and dynamic gestures, using simple RGB vision (without depth sensing). This feature makes it suitable for inexpensive human-robot interaction in social or industrial settings. We employ a pose-driven spatial attention strategy, which guides our proposed Static and Dynamic gestures Network - StaDNet. From the image of the human upper body, we estimate his/her depth, along with the region-of-interest around his/her hands. The Convolutional Neural Network in StaDNet is fine-tuned on a background-substituted hand gestures dataset. It is utilized to detect 10 static gestures for each hand as well as to obtain the hand image-embeddings. These are subsequently fused with the augmented pose vector and then passed to the stacked Long Short-Term Memory blocks. Thus, human-centred frame-wise information from the augmented pose vector and from the left/right hands image-embeddings are aggregated in time to predict the dynamic gestures of the performing person. In a number of experiments, we show that the proposed approach surpasses the state-of-the-art results on the large-scale Chalearn 2016 dataset. Moreover, we transfer the knowledge learned through the proposed methodology to the Praxis gestures dataset, and the obtained results also outscore the state-of-the-art on this dataset.

Computer Vision and Pattern Recognition,Human-Computer Interaction,Machine Learning,Robotics
Multistage Spatial Attention-Based Neural Network for Hand Gesture Recognition

Abu Saleh Musa Miah,Md. Al Mehedi Hasan,Jungpil Shin,Yuichi Okuyama,Yoichi Tomioka

DOI: https://doi.org/10.3390/computers12010013

2023-01-05

Computers

Abstract:The definition of human-computer interaction (HCI) has changed in the current year because people are interested in their various ergonomic devices ways. Many researchers have been working to develop a hand gesture recognition system with a kinetic sensor-based dataset, but their performance accuracy is not satisfactory. In our work, we proposed a multistage spatial attention-based neural network for hand gesture recognition to overcome the challenges. We included three stages in the proposed model where each stage is inherited the CNN; where we first apply a feature extractor and a spatial attention module by using self-attention from the original dataset and then multiply the feature vector with the attention map to highlight effective features of the dataset. Then, we explored features concatenated with the original dataset for obtaining modality feature embedding. In the same way, we generated a feature vector and attention map in the second stage with the feature extraction architecture and self-attention technique. After multiplying the attention map and features, we produced the final feature, which feeds into the third stage, a classification module to predict the label of the correspondent hand gesture. Our model achieved 99.67%, 99.75%, and 99.46% accuracy for the senz3D, Kinematic, and NTU datasets.
Multimodal Spatiotemporal Feature Map for Dynamic Gesture Recognition

Xiaorui Zhang,Xianglong Zeng,Wei Sun,Yongjun Ren,Tong Xu

DOI: https://doi.org/10.32604/csse.2023.035119

IF: 4.397

2023-01-01

Computer Systems Science and Engineering

Abstract:Gesture recognition technology enables machines to read human gestures and has significant application prospects in the fields of human-computer interaction and sign language translation. Existing researches usually use convolutional neural networks to extract features directly from raw gesture data for gesture recognition, but the networks are affected by much interference information in the input data and thus fit to some unimportant features. In this paper, we proposed a novel method for encoding spatio-temporal information, which can enhance the key features required for gesture recognition, such as shape, structure, contour, position and hand motion of gestures, thereby improving the accuracy of gesture recognition. This encoding method can encode arbitrarily multiple frames of gesture data into a single frame of the spatio-temporal feature map and use the spatio-temporal feature map as the input to the neural network. This can guide the model to fit important features while avoiding the use of complex recurrent network structures to extract temporal features. In addition, we designed two sub-networks and trained the model using a sub-network pre-training strategy that trains the sub-networks first and then the entire network, so as to avoid the sub-networks focusing too much on the information of a single category feature and being overly influenced by each other’s features. Experimental results on two public gesture datasets show that the proposed spatio-temporal information encoding method achieves advanced accuracy.

computer science, theory & methods, hardware & architecture

The diffusion coefficient of a swollen microgel particle.

RGC: Reliable Gesture Classification Via Wearables Using GANs-Based Data Augmentation.

Multi-Scale Attention 3D Convolutional Network for Multimodal Gesture Recognition

Regional Attention with Architecture-Rebuilt 3D Network for RGB-D Gesture Recognition

Multi-Task and Multi-Modal Learning for RGB Dynamic Gesture Recognition

Short-Term Temporal Convolutional Networks for Dynamic Hand Gesture Recognition

Dynamic Gesture Recognition Method Based on Improved R(2+1)D

Gesture recognition based on deep deformable 3D convolutional neural networks

Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition

Gesture Recognition with a 3-D Accelerometer

Dynamic Gesture Recognition Algorithm Based on 3D Convolutional Neural Network

Dynamic Gesture Recognition Based on Two-Scale 3-D-ConvNeXt

Dynamic Hand Gesture Recognition Based on 3D Hand Pose Estimation for Human-Robot Interaction

AL-MobileNet: a Novel Model for 2D Gesture Recognition in Intelligent Cockpit Based on Multi-Modal Data

Large-scale Isolated Gesture Recognition Using Convolutional Neural Networks

GesID: 3D Gesture Authentication Based on Depth Camera and One-Class Classification

Necessity for including quality of life diagnostics and therapy in routine oncological care: Evaluation by methods of clinical guideline implementation.

FGDSNet: A Lightweight Hand Gesture Recognition Network for Human Robot Interaction

A Deep Learning Framework for Recognizing both Static and Dynamic Gestures

Multistage Spatial Attention-Based Neural Network for Hand Gesture Recognition

Multimodal Spatiotemporal Feature Map for Dynamic Gesture Recognition