Abstract:The traditional human-computer interaction is mainly through the mouse, keyboard, remote control, and other peripheral equipment electromagnetic signal transmission. This paper aims to build a visual human-computer interaction system through a series of deep learning and machine vision models, so that people can achieve complete human-computer interaction only through the camera and screen. The established visual human-computer interaction system mainly includes the function modes of three basic peripherals in human-computer interaction: keyboard, mouse (X-Y position indicator), and remote control. The convex hull method was used to switch between these three modes. After issuing the mode command, Gaussian mixture was used to quickly identify the moving human body to narrow the scope of our image processing. Subsequently, finger detection in human body was realized based on the Faster-RCNN-ResNET50-FPN model structure, and realized the function of moving mouse and keyboard through the relationship between different fingers. At the same time, the recognition of human body posture was done by using MediaPipe BlazePose, and the action classification models were established through the Angle between body movements so as to realize the control function of remote control. In order to ensure the real-time performance of the interactive system, according to the characteristics of different data processing processes, CPU and GPU computing power resources are used to cross-process images to ensure the real-time performance. The recognition accuracy of the human-computer interaction system is above 0.9 for the key feature points of human body, and above 0.87 for the recognition accuracy of four kinds of command actions. It is hoped that vision-based human-computer interaction will become a widely used interaction mode in the future.

Hand Motion Based Human Computer Interaction Using 3D Convolutional Neural Network

An Efficient Human Computer Interaction through Hand Gesture Using Deep Convolutional Neural Network

Gesture Recognition Using Inertial Sensors with 1D Convolutional Neural Network

Visual Hand Gesture Recognition With Convolution Neural Network

Smart Home Automation-Based Hand Gesture Recognition Using Feature Fusion and Recurrent Neural Network

Real-Time Hand Gesture Spotting and Recognition Using RGB-D Camera and 3D Convolutional Neural Network

Recognition of Hand Gesture Image Using Deep Convolutional Neural Network

A Secure Approach for Human Computer Interaction Using Human Hand Action

Deep learning based Hand gesture recognition system and design of a Human-Machine Interface

3D dynamic hand gestures recognition using the Leap Motion sensor and convolutional neural networks

Multistage Spatial Attention-Based Neural Network for Hand Gesture Recognition

An Integrative Approach to Robust Hand Detection Using CPM-YOLOv3 and RGBD Camera in Real Time.

3D Convolutional Neural Network for Human Behavior Analysis in Intelligent Sensor Network

Dynamic Hand Gesture Recognition Based on 3D Hand Pose Estimation for Human-Robot Interaction

Short-Term Temporal Convolutional Networks for Dynamic Hand Gesture Recognition

Fast and Accurate 3D Hand Pose Estimation via Recurrent Neural Network for Capturing Hand Articulations

Human Motion Gesture Recognition Based on Computer Vision

Dynamic Gesture Recognition Algorithm Based on 3D Convolutional Neural Network

Hand Gesture Classification Based on Forearm Ultrasound Video Snippets Using 3D Convolutional Neural Networks

A Visual Human-Computer Interaction System Based on Hybrid Visual Model

Recognition of Social Touch Gestures Using 3d Convolutional Neural Networks