Abstract:The traditional human-computer interaction is mainly through the mouse, keyboard, remote control, and other peripheral equipment electromagnetic signal transmission. This paper aims to build a visual human-computer interaction system through a series of deep learning and machine vision models, so that people can achieve complete human-computer interaction only through the camera and screen. The established visual human-computer interaction system mainly includes the function modes of three basic peripherals in human-computer interaction: keyboard, mouse (X-Y position indicator), and remote control. The convex hull method was used to switch between these three modes. After issuing the mode command, Gaussian mixture was used to quickly identify the moving human body to narrow the scope of our image processing. Subsequently, finger detection in human body was realized based on the Faster-RCNN-ResNET50-FPN model structure, and realized the function of moving mouse and keyboard through the relationship between different fingers. At the same time, the recognition of human body posture was done by using MediaPipe BlazePose, and the action classification models were established through the Angle between body movements so as to realize the control function of remote control. In order to ensure the real-time performance of the interactive system, according to the characteristics of different data processing processes, CPU and GPU computing power resources are used to cross-process images to ensure the real-time performance. The recognition accuracy of the human-computer interaction system is above 0.9 for the key feature points of human body, and above 0.87 for the recognition accuracy of four kinds of command actions. It is hoped that vision-based human-computer interaction will become a widely used interaction mode in the future.

Study on the Model of Control Center Large Screen's HCI Based on MMI

Research of Key Technical Issues on Handset MMI Automated Testing System

COMTIS: Customizable Touchless Interaction System for Large Screen Visualization

A Survey of Multimodal Human-Machine Interface.

Human-machine Interface for a Master-Slave Hydraulic Manipulator with Vision Enhancement and Auditory Feedback

Man-machine Interface Design for Power Quality Monitor

Design and Development of a Gesture-Based Interaction System for Multi-Projector Tiled Display Wall

A Multimodal Human-Computer Interaction System and Its Application in Smart Learning Environments

A Visual Human-Computer Interaction System Based on Hybrid Visual Model

Vehicle Human-Machine Interaction Interface Evaluation Method Based on Eye Movement and Finger Tracking Technology

Navigating High-Resolution Image Visualization on a Large Display using Multimodal Interaction

Research on the Usability Evaluation Model of Human-Computer Interaction for Command and Control System

Development and Application of Integrated Human Machine Interaction Simulation System

Integrated Design of System Display and Procedural Display in Advanced NPP Control Rooms

Research on Man-Machine Interaction of Handheld Mobile Computing

An Intelligent Platform for the Multimedia Instruction Environment Based on i-Control, i-Management and i-View

A large display-based approach supporting natural user interaction in virtual reality environment

Is Silent eHMI Enough? A Passenger-Centric Study on Effective eHMI for Autonomous Personal Mobility Vehicles in the Field

Designing Autonomous Driving HMI System: Interaction Need Insight and Design Tool Study.

Design of a new human-computer interactive device for projection display

Parallel Orientation Assistant, a Vehicle System Based on Voice Interaction and Multi-screen Interaction.