MISAR: A Multimodal Instructional System with Augmented Reality

Jing Bi,Nguyen Manh Nguyen,Ali Vosoughi,Chenliang Xu

2023-10-18

Abstract:Augmented reality (AR) requires the seamless integration of visual, auditory, and linguistic channels for optimized human-computer interaction. While auditory and visual inputs facilitate real-time and contextual user guidance, the potential of large language models (LLMs) in this landscape remains largely untapped. Our study introduces an innovative method harnessing LLMs to assimilate information from visual, auditory, and contextual modalities. Focusing on the unique challenge of task performance quantification in AR, we utilize egocentric video, speech, and context analysis. The integration of LLMs facilitates enhanced state estimation, marking a step towards more adaptive AR systems. Code, dataset, and demo will be available at <a class="link-external link-https" href="https://github.com/nguyennm1024/misar" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to improve the quantitative evaluation of task execution by integrating visual, auditory and language modalities and using large - language models (LLMs) in an augmented reality (AR) environment. Specifically, the paper proposes a multimodal system MISAR (Multimodal Instructional System with Augmented Reality), which can enhance state estimation through first - person video, voice and context analysis, thereby improving the adaptability and intelligence level of AR systems. The main contributions of the paper are as follows:

1. **Introducing a comprehensive method**: Use AI technology to help people complete tasks more accurately and efficiently.
2. **Using large - language models (LLMs) to improve video captions**: By enhancing the quality of video captions, improve the ability of scene understanding without the need for domain - specific data training.

The paper elaborates on how to integrate technologies such as automatic speech recognition (ASR), text - to - speech (TTS), video - to - text conversion, etc., combined with large - language models such as GPT3.5 - Turbo, to realize a user - friendly interaction system that can seamlessly switch between multiple modalities and provide real - time guidance and error - correction functions. In addition, the paper also shows the effectiveness of this method through experiments and demonstrates the practical value of the system through actual demonstrations.