Abstract:The traditional human-computer interaction is mainly through the mouse, keyboard, remote control, and other peripheral equipment electromagnetic signal transmission. This paper aims to build a visual human-computer interaction system through a series of deep learning and machine vision models, so that people can achieve complete human-computer interaction only through the camera and screen. The established visual human-computer interaction system mainly includes the function modes of three basic peripherals in human-computer interaction: keyboard, mouse (X-Y position indicator), and remote control. The convex hull method was used to switch between these three modes. After issuing the mode command, Gaussian mixture was used to quickly identify the moving human body to narrow the scope of our image processing. Subsequently, finger detection in human body was realized based on the Faster-RCNN-ResNET50-FPN model structure, and realized the function of moving mouse and keyboard through the relationship between different fingers. At the same time, the recognition of human body posture was done by using MediaPipe BlazePose, and the action classification models were established through the Angle between body movements so as to realize the control function of remote control. In order to ensure the real-time performance of the interactive system, according to the characteristics of different data processing processes, CPU and GPU computing power resources are used to cross-process images to ensure the real-time performance. The recognition accuracy of the human-computer interaction system is above 0.9 for the key feature points of human body, and above 0.87 for the recognition accuracy of four kinds of command actions. It is hoped that vision-based human-computer interaction will become a widely used interaction mode in the future.

Summary of the Research on the Interaction of Tightly Coupled Physical Human-machine System

The Design and Implementation of Natural Human-Robot Interaction System Based on Kinect Sensor

Function Features of the Human-Machine Interaction in the Computer Pan-Vision System

The Interaction Design of 3D Virtual Humans: A Survey

Design of Human-Machine Integration System to Meet Diverse Interactive Tasks

Human-centric Application in Cyber-Physical System: an Inertial-based Motion Capture and Recognition System

Application of human-computer interaction technology in rehabilitation treatment of mental and nervous system diseases

A Survey of Multimodal Human-Machine Interface.

Soft Human–machine Interfaces: Design, Sensing and Stimulation

Review of Sensory Feedback Simulation Methods in Multi-modal Human-computer Interaction

A Multi‐Module Sensing and Bi‐Directional HMI Integrating Interaction, Recognition, and Feedback for Intelligent Robots

A Solution of Human-Computer Remote Interaction with Tactile Feedback

On Human-Machine Interaction of Bionic Prosthetic Hand with Sensory Feedback

A Visual Human-Computer Interaction System Based on Hybrid Visual Model

Research on information product design method based on the man-machine coupling

Research and Prospects of Virtual Reality Systems Applying Exoskeleton Technology

Development and Application of Integrated Human Machine Interaction Simulation System

Recent Progress of Tactile and Force Sensors for Human-Machine Interaction

Design and Optimization of Human-Machine Interaction Wearable Device for Lower Limb Rehabilitation Exoskeleton

Research on Man-Machine Interaction of Handheld Mobile Computing

The Research of "mechanical Virtual Human of China" and Its Application