Abstract:In this study, the design of a Digital-twin human-machine interface sensor (DT-HMIS) is proposed. This is a digital-twin sensor (DT-Sensor) that can meet the demands of human-machine automation collaboration in Industry 5.0. The DT-HMIS allows users/patients to add, modify, delete, query, and restore their previously memorized DT finger gesture mapping model and programmable logic controller (PLC) logic program, enabling the operation or access of the programmable controller input-output (I/O) interface and achieving the extended limb collaboration capability of users/patients. The system has two main functions: the first is gesture-encoded virtual manipulation, which indirectly accesses the PLC through the DT mapping model to complete control of electronic peripherals for extension-limbs ability by executing logic control program instructions. The second is gesture-based virtual manipulation to help non-verbal individuals create special verbal sentences through gesture commands to improve their expression ability. The design method uses primitive image processing and eight-way dual-bit signal processing algorithms to capture the movement of human finger gestures and convert them into digital signals. The system service maps control instructions by observing the digital signals of the DT-HMIS and drives motion control through mechatronics integration or speech synthesis feedback to express the operation requirements of inconvenient work or complex handheld physical tools. Based on the human-machine interface sensor of DT computer vision, it can reflect the user's command status without the need for additional wearable devices and promote interaction with the virtual world. When used for patients, the system ensures that the user's virtual control is mapped to physical device control, providing the convenience of independent operation while reducing caregiver fatigue. This study shows that the recognition accuracy can reach 99%, demonstrating practicality and application prospects. In future applications, users/patients can interact virtually with other peripheral devices through the DT-HMIS to meet their own interaction needs and promote industry progress.

Finger–Coding Intelligent Human–Machine Interaction System Based on All–Fabric Ionic Capacitive Pressure Sensors

A User-Friendly Packaging Solution for Silicon-Based Pressure Die to Develop Prosthetic Tactile Sensing

A Multipurpose Human–Machine Interface via 3D-Printed Pressure-Based Force Myography

Noncontact Human-machine Interaction for Air-writing Based on Piezoelectric Micromachined Ultrasonic Transducers

Triboelectric-Inertial Sensing Glove Enhanced by Charge-Retained Strategy for Human-Machine Interaction.

Noncontact human-machine interaction for air-writing with ultrasound technology

A Portable Somatosensory Manipulator System Based on Graphene Ink/Paper Film Piezoresistive Sensors for Human–Computer Interaction

Triboelectric bending sensor based smart glove towards intuitive multi-dimensional human-machine interfaces

Fingerprint-inspired Dual-mode Pressure Sensor for Robotic Static and Dynamic Perception

Noncontact Human-Machine Interface Using Complementary Information Fusion Based on MEMS and Triboelectric Sensors

A Flexible Dual-Modal Sensing System for Synchronous Pressure and Inertial Monitoring of Finger Movement

Design of Digital-Twin Human-Machine Interface Sensor with Intelligent Finger Gesture Recognition

Design of a Capacitive Tactile Sensor Array System for Human–Computer Interaction

Self‐Powered, Soft and Breathable Human–Machine Interface Based on Piezoelectric Sensors

A Flexible Iontronic Capacitive Sensing Array for Hand Gesture Recognition Using Deep Convolutional Neural Networks

An Intelligent Glove System for Real‐Time Multiple Electrical Signal Acquisition and Display

Programmable and Ultrasensitive Haptic Interfaces Enabling Closed‐Loop Human–Machine Interactions

High‐accuracy Dynamic Gesture Recognition: A Universal and Self‐adaptive Deep‐learning‐assisted System Leveraging High‐performance Ionogels‐based Strain Sensors

Anatomically Designed Triboelectric Wristbands with Adaptive Accelerated Learning for Human–Machine Interfaces

High‐Performance Hydrogel Sensors Enabled Multimodal And Accurate Human–Machine Interaction System for Active Rehabilitation

Flexible Capacitive Three-Dimensional Force Sensor for Hand Motion Capture and Handwriting Recognition