A Multi-modal Virtual-Real Fusion System for Multi-task Human-Computer Interaction.
Xinlei Zhang,Jiahui Yu,Yuxiang Sun,Min Li,Yang Song,Xianzhong Zhou
DOI: https://doi.org/10.1109/icnsc55942.2022.10004097
2022-01-01
Abstract:Due to the complexity of the task and the diversification of the scene, the traditional interactive control method can no longer meet the requirements of users. To solve this problem, a multi-modal virtual-real fusion system for multi-task human-computer interaction is proposed in this paper, which integrates eye movement, gesture and voice. Frist, aiming at the phenomenon of multi-task and multi-modal, a task-modal matching model is established. Then, the task-modal matching model is abstracted into a multi-objective optimization problem, and a method for solving this problem is designed and a matching scheme is successfully obtained. Meanwhile, the construction of the system is completed for the virtual-real fusion environment, and the control of unmanned car and virtual car is realized. The system can carry out multi-modal interaction and complete multiple tasks in real scene, virtual scene, parallel system and virtual-real fusion scene. Finally, the experiment proves the stability and reliability of the system.