Abstract:In recent years, virtual reality arenas have become increasingly popular for quantifying visual behaviours. By using the actions of a constrained animal to control the visual scenery, the animal perceives that it is moving through a virtual world. Importantly, as the animal is constrained in space, behavioural quantification is facilitated. Furthermore, using computer‐generated visual scenery allows for identification of visual triggers of behaviour. We created a novel virtual reality arena combining machine vision with the gaming engine Unity. For tethered flight, we enhanced an existing multi‐modal virtual reality arena, MultiMoVR, but tracked wing movements using DeepLabCut‐live (DLC‐live). For tethered walking animals, we used FicTrac to track the motion of a trackball. In both cases, real‐time tracking was interfaced with Unity to control the location and rotation of the tethered animal's avatar in the virtual world. We developed a user‐friendly Unity Editor interface, CAVE, to simplify experimental design and data storage without the need for coding. We show that both the DLC‐live‐Unity and the FicTrac‐Unity configurations close the feedback loop effectively and quickly. We show that closed‐loop feedback reduces behavioural artefacts exhibited by walking crabs in open‐loop scenarios, and that flying Eristalis tenax hoverflies navigate towards virtual flowers in closed loop. We show examples of how the CAVE interface can enable experimental sequencing control including use of avatar proximity to virtual objects of interest. Our results show that combining Unity with machine vision tools provides an easy and flexible virtual reality environment that can be readily adjusted to new experiments and species. This can be implemented programmatically in Unity, or by using our new tool CAVE, which allows users to design new experiments without additional programming. We provide resources for replicating experiments and our interface CAVE via GitHub, together with user manuals and instruction videos, for sharing with the wider scientific community.

VR-GPT: Visual Language Model for Intelligent Virtual Reality Applications

Virtual Reality and Language Models, a New Frontier in Learning

Supporting Text Entry in Virtual Reality with Large Language Models

Exploring a GPT-based large language model for variable autonomy in a VR-based human-robot teaming simulation

Exploring Large Language Models to Facilitate Variable Autonomy for Human-Robot Teaming

ELLMA-T: an Embodied LLM-agent for Supporting English Language Learning in Social VR

Voice2Action: Language Models as Agent for Efficient Real-Time Interaction in Virtual Reality

Large Language Model-assisted Speech and Pointing Benefits Multiple 3D Object Selection in Virtual Reality

Expanding Frozen Vision-Language Models without Retraining: Towards Improved Robot Perception

Takeaways from Applying LLM Capabilities to Multiple Conversational Avatars in a VR Pilot Study

Immersed in my Ideas: Using Virtual Reality and Multimodal Interactions to Visualize Users' Ideas and Thoughts

Evaluating Usability and Engagement of Large Language Models in Virtual Reality for Traditional Scottish Curling

LLMR: Real-time Prompting of Interactive Worlds using Large Language Models

An NLP-based system for modulating virtual experiences using speech instructions

Integrating Large Language Models with Multimodal Virtual Reality Interfaces to Support Collaborative Human-Robot Construction Work

Hands-Free VR

From Concept to Manufacturing: Evaluating Vision-Language Models for Engineering Design

AI-Powered VR for Enhanced Learning Compared to Traditional Methods

DataliVR: Transformation of Data Literacy Education through Virtual Reality with ChatGPT-Powered Enhancements

Combining Unity with machine vision to create low latency, flexible and simple virtual realities

Design and Optimization of English-Speaking Teaching Model Using Virtual Reality Technology