Abstract:The proposed research introduces an innovative Virtual Reality (VR) and Large Language Model (LLM) architecture to enhance the learning process across diverse educational contexts, ranging from school to industrial settings. everaging the capabilities of LLMs and Retrieval-Augmented Generation (RAG), the architecture centers around an immersive VR application. This application empowers students of all backgrounds to interactively engage with their environment by posing questions and receiving informative responses in text format and with visual hints in VR, thereby fostering a dynamic learning experience. LLMs with RAG act as the backbones of this architecture, facilitating the integration of private or domain-specific data into the learning process. By seamlessly connecting various data sources through data connectors, RAG overcomes the challenge of disparate and siloed information repositories, including APIs, PDFs, SQL databases, and more. The data indexes provided by RAG solutions further streamline this process by structuring the ingested data into formats optimized for consumption by LLMs. An empirical study was conducted to evaluate the effectiveness of this VR and LLM architecture. Twenty participants, divided into Experimental and Control groups, were selected to assess the impact on their learning process. The Experimental group utilized the immersive VR application, which allowed interactive engagement with the educational environment, while the Control group followed traditional learning methods. The study revealed significant improvements in learning outcomes for the Experimental group, demonstrating the potential of integrating VR and LLMs in enhancing comprehension and engagement in learning contexts. This study presents an innovative approach that capitalizes on the synergy between LLMs and immersive VR technology, opening avenues for a transformative learning experience that transcends traditional boundaries and empowers learners across a spectrum of educational landscapes.

Voice2Action: Language Models as Agent for Efficient Real-Time Interaction in Virtual Reality

Learning Personalized Agent for Real-Time Face-to-Face Interaction in VR

Let's Give a Voice to Conversational Agents in Virtual Reality

Supporting Text Entry in Virtual Reality with Large Language Models

ELLMA-T: an Embodied LLM-agent for Supporting English Language Learning in Social VR

Human and LLM-Based Voice Assistant Interaction: An Analytical Framework for User Verbal and Nonverbal Behaviors

Takeaways from Applying LLM Capabilities to Multiple Conversational Avatars in a VR Pilot Study

Multimodal Human-Autonomous Agents Interaction Using Pre-Trained Language and Visual Foundation Models

User Interaction Patterns and Breakdowns in Conversing with LLM-Powered Voice Assistants

Large Language Model-assisted Speech and Pointing Benefits Multiple 3D Object Selection in Virtual Reality

Virtual Reality and Language Models, a New Frontier in Learning

Improving Agent Interactions in Virtual Environments with Language Models

VR-GPT: Visual Language Model for Intelligent Virtual Reality Applications

The influence of persona and conversational task on social interactions with a LLM-controlled embodied conversational agent

Integrating Large Language Models with Multimodal Virtual Reality Interfaces to Support Collaborative Human-Robot Construction Work

Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case

LLMR: Real-time Prompting of Interactive Worlds using Large Language Models

3D-VLA: A 3D Vision-Language-Action Generative World Model

Cooperation on the Fly: Exploring Language Agents for Ad Hoc Teamwork in the Avalon Game

Behavioral Analysis of Vision-and-Language Navigation Agents

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents