MAG: a smart gloves system based on multimodal fusion perception

Hong Cui,Zhiquan Feng,Jinglan Tian,Dehui Kong,Zishuo Xia,Weina Li
DOI: https://doi.org/10.1007/s42486-023-00138-5
2023-09-11
CCF Transactions on Pervasive Computing and Interaction
Abstract:The value and significance of intelligent experiments have gained widespread consensus and formed an industry chain in the education field. However, the interaction issues in such systems have severely constrained their extensive application. These issues manifest in weak operational feedback, problems with traditional interactive devices such as obstructed views and inability to observe subtle experimental phenomena at close range. The most prominent challenge is the inability of existing virtual experiments to accurately perceive user intentions. Therefore, in response to the aforementioned challenging problems in intelligent experiment interaction tools, this paper proposes an overall approach to design a multimodal fusion perception-based Smart gloves system for middle school experiments. It adopts a combination of virtual and real experiments, utilizing the Smart gloves to facilitate the experiment process. The paper designs the hardware structure of the Smart gloves and proposes the YSA algorithm and ICAM algorithm to capture user multimodal intentions, constructing the MAG algorithm to achieve multimodal fusion comprehension of user intentions at the decision-making level. This enhances the system's ability to accurately comprehend and analyze user intentions, thereby strengthening the Smart gloves's intent understanding capability. Experimental results demonstrate that the designed Smart gloves can promptly infer users' experimental intentions and provide corresponding feedback. The experiments further indicate that the developed Smart gloves interaction tool exhibits unique advantages in addressing challenging technical issues faced by existing intelligent experiment tools and traditional data gloves, offering encouraging potential application prospects.
What problem does this paper attempt to address?