VAEnvGen: A Real-Time Virtual Agent Environment Generation System Based on Large Language Models
Jingyu Wu Pengchen Chen Shi Chen Xiang Wei Lingyun Sun a College of Computer Science and Technology,Zhejiang University,Hangzhou,Chinab Zhejiang-Singapore Innovation and AI Joint Research Lab,Hangzhou,ChinaJingyu Wu is a Ph.D. attached to the College of Computer Science and Technology,Zhejiang University. With a background in HCI and CV,his research focuses on multi-modal SIA development and human-AI interaction.Pengchen Chen is an undergraduate currently working as a research assistant at the International Design Institute of Zhejiang University. His research interests focus on human-AI interaction and virtual agents.Chen Shi is currently an assistant professor in Industrial Design Department,Zhejiang University. Her research interests lie in Information and Interaction Design,Visual Design Computing,and Design cognition. She has published many research papers in various reputable journals and conference proceedings.Wei Xiang is a lecturer in the Industrial Design Department,Zhejiang University. He received his PhD degree in Digital Art and Design. His research lies in design intelligence and human–computer interaction.Lingyun Sun is a professor at the College of Computer Science and Technology,Zhejiang University. He is the deputy director of the International Design Institute of Zhejiang University. His research interests include human-computer interaction,creative intelligence,and information and interaction design.
DOI: https://doi.org/10.1080/10447318.2024.2387398
IF: 4.92
2024-08-23
International Journal of Human-Computer Interaction
Abstract:Environment plays an important role in non-verbal communication for human-virtual agent interaction. Existing research explores the influence of an agent's appearance and attributes to enhance human-virtual agent communication. However, there is no common practice for dynamically adjusting the surrounding environments of the virtual agent. In this paper, we introduce a real-time virtual agent environment generation system (VAEnvGen), which contributes to the field by enhancing users' content perception and improving task performance through dynamic environment adjustment. The system dynamically analyzes both the appropriate communication environment and filters the key information according to the current context. Leveraging Large Language Models, it generates a pseudo-3D background space to create an engaging atmosphere and a dynamic foreground content space for vivid key information display, thereby significantly enhancing content perception. For widespread adoption and flexibility, VAEnvGen is developed as a web application. We further evaluate the impact of VAEnvGen on content perception, user attention, and subjective satisfaction through a mixed-design user study with 50 participants. Quantitative and qualitative results reveal significant improvements in content perception, task completion time, and user satisfaction when using VAEnvGen. The system effectively redistributes user attention from subtitles and the virtual agent itself to the dynamically generated background and key foreground information, leading to a more immersive and less fatiguing user experience.
computer science, cybernetics,ergonomics