Abstract:In recent years, virtual reality arenas have become increasingly popular for quantifying visual behaviours. By using the actions of a constrained animal to control the visual scenery, the animal perceives that it is moving through a virtual world. Importantly, as the animal is constrained in space, behavioural quantification is facilitated. Furthermore, using computer‐generated visual scenery allows for identification of visual triggers of behaviour. We created a novel virtual reality arena combining machine vision with the gaming engine Unity. For tethered flight, we enhanced an existing multi‐modal virtual reality arena, MultiMoVR, but tracked wing movements using DeepLabCut‐live (DLC‐live). For tethered walking animals, we used FicTrac to track the motion of a trackball. In both cases, real‐time tracking was interfaced with Unity to control the location and rotation of the tethered animal's avatar in the virtual world. We developed a user‐friendly Unity Editor interface, CAVE, to simplify experimental design and data storage without the need for coding. We show that both the DLC‐live‐Unity and the FicTrac‐Unity configurations close the feedback loop effectively and quickly. We show that closed‐loop feedback reduces behavioural artefacts exhibited by walking crabs in open‐loop scenarios, and that flying Eristalis tenax hoverflies navigate towards virtual flowers in closed loop. We show examples of how the CAVE interface can enable experimental sequencing control including use of avatar proximity to virtual objects of interest. Our results show that combining Unity with machine vision tools provides an easy and flexible virtual reality environment that can be readily adjusted to new experiments and species. This can be implemented programmatically in Unity, or by using our new tool CAVE, which allows users to design new experiments without additional programming. We provide resources for replicating experiments and our interface CAVE via GitHub, together with user manuals and instruction videos, for sharing with the wider scientific community.

Magical Experience with Full-body Action

Exploitation of Multiplayer Interaction and Development of Virtual Puppetry Storytelling Using Gesture Control and Stereoscopic Devices.

Responsive Action Generation By Physically-Based Motion Retrieval And Adaptation

Exploitation Of Novel Multiplayer Gesture-Based Interaction And Virtual Puppetry For Digital Storytelling To Develop Children'S Narrative Skills

Building Virtual Entertainment Environment with Tiled Display Wall and Motion Tracking.

A Motion-based User Interface for the Control of Virtual Humans Performing Sports.

Interactive Animation of Virtual Characters: Application to Virtual Kung-Fu Fighting

Interpretation of Tactile Sensation using an Anthropomorphic Finger Motion Interface to Operate a Virtual Avatar

Implementation of the Interactive Gestures of Virtual Avatar Based on a Multi-user Virtual Learning Environment

The Wanderings of Odysseus in 3D Scenes

Motion-Based Perceptual User Interface

Physical World to Virtual Reality – Motion Capture Technology in Dance Creation

H.: Magic Land: a 3D Human Capture Mixed Reality System for Museum Experiences

Touch-Space: Mixed Reality Game Space Based on Ubiquitous, Tangible, and Social Computing

Perception of Walking Self-body Avatar Enhances Virtual-walking Sensation

Automatic Generation of Human Animation Based on Motion Programming.

Automatic Generation of Human Animation Based on Motion Programming: Virtual Humans and Social Agents

MagicAvatar: Multimodal Avatar Generation and Animation

Combining Unity with machine vision to create low latency, flexible and simple virtual realities

Neural3Points: Learning to Generate Physically Realistic Full-body Motion for Virtual Reality Users

A Full Body Avatar-Based Telepresence System for Dissimilar Spaces