Abstract:The traditional human-computer interaction is mainly through the mouse, keyboard, remote control, and other peripheral equipment electromagnetic signal transmission. This paper aims to build a visual human-computer interaction system through a series of deep learning and machine vision models, so that people can achieve complete human-computer interaction only through the camera and screen. The established visual human-computer interaction system mainly includes the function modes of three basic peripherals in human-computer interaction: keyboard, mouse (X-Y position indicator), and remote control. The convex hull method was used to switch between these three modes. After issuing the mode command, Gaussian mixture was used to quickly identify the moving human body to narrow the scope of our image processing. Subsequently, finger detection in human body was realized based on the Faster-RCNN-ResNET50-FPN model structure, and realized the function of moving mouse and keyboard through the relationship between different fingers. At the same time, the recognition of human body posture was done by using MediaPipe BlazePose, and the action classification models were established through the Angle between body movements so as to realize the control function of remote control. In order to ensure the real-time performance of the interactive system, according to the characteristics of different data processing processes, CPU and GPU computing power resources are used to cross-process images to ensure the real-time performance. The recognition accuracy of the human-computer interaction system is above 0.9 for the key feature points of human body, and above 0.87 for the recognition accuracy of four kinds of command actions. It is hoped that vision-based human-computer interaction will become a widely used interaction mode in the future.

The Detection,Location and Movement Analysis of Targets in Visual Interaction System

Research and Realization on Moving Object Detection and Tracking Methods of Human

2D Motion Detection Bounded Hand 3D Trajectory Tracking and Gesture Recognition under Complex Background

A Penetrable Interactive 3D Display Based on Motion Recognition （invited Paper）

Interactive Progressive Operator-based IBM System

Vision-Based Moving Objects Detection with Background Modeling

Building Virtual Entertainment Environment with Tiled Display Wall and Motion Tracking.

2D Tracking Algorithm Performance Testing Based on Three- Dimensional Visual Platform

An Interactive Control System for 3d Display Using Hidden Markov Model

Virtual-Physical Interaction System

3D Intuitive Gesture Interaction via Motion Sensing

Stereo vision based 3D game interface

DESIGN AND INPLEMENTATION OF DYNAMIC RESPONSE MECHANISM BETWEEN 2D DIGITAL MAP AND 3D VISUALIZATION SCENE

Interactive Photo-Browsing System Based on Moving Target Detection

A Visual Human-Computer Interaction System Based on Hybrid Visual Model

Study on Detection System of the Planar Parallel Micro-motion Worktable's Location and Posture

3D Indoor Environment Modeling and Detection of Moving Object and Scene Understanding

Design of Movements Detection and Recognition System Based on Video Identification Technology

A Model-Based Analysis-Design Approach for Virtual Binocular Vision System with Application to Vision-Based Tactile Sensors

Design and Implementation of a Real-time Target Detection and Tracking System

Study on Modeling of the Common Three-Dimension Visual System of Railway Simulation