Abstract:In recent years, virtual reality arenas have become increasingly popular for quantifying visual behaviours. By using the actions of a constrained animal to control the visual scenery, the animal perceives that it is moving through a virtual world. Importantly, as the animal is constrained in space, behavioural quantification is facilitated. Furthermore, using computer‐generated visual scenery allows for identification of visual triggers of behaviour. We created a novel virtual reality arena combining machine vision with the gaming engine Unity. For tethered flight, we enhanced an existing multi‐modal virtual reality arena, MultiMoVR, but tracked wing movements using DeepLabCut‐live (DLC‐live). For tethered walking animals, we used FicTrac to track the motion of a trackball. In both cases, real‐time tracking was interfaced with Unity to control the location and rotation of the tethered animal's avatar in the virtual world. We developed a user‐friendly Unity Editor interface, CAVE, to simplify experimental design and data storage without the need for coding. We show that both the DLC‐live‐Unity and the FicTrac‐Unity configurations close the feedback loop effectively and quickly. We show that closed‐loop feedback reduces behavioural artefacts exhibited by walking crabs in open‐loop scenarios, and that flying Eristalis tenax hoverflies navigate towards virtual flowers in closed loop. We show examples of how the CAVE interface can enable experimental sequencing control including use of avatar proximity to virtual objects of interest. Our results show that combining Unity with machine vision tools provides an easy and flexible virtual reality environment that can be readily adjusted to new experiments and species. This can be implemented programmatically in Unity, or by using our new tool CAVE, which allows users to design new experiments without additional programming. We provide resources for replicating experiments and our interface CAVE via GitHub, together with user manuals and instruction videos, for sharing with the wider scientific community.

Moculus: an immersive virtual reality system for mice incorporating stereo vision

Multichannel Fiber-Based Simultaneous Neural Stimulation and Signal Readout in Vivo

MouseGoggles: an immersive virtual reality headset for mouse neuroscience and behavior

An Open-Source Virtual Reality System for the Measurement of Spatial Learning in Head-Restrained Mice

A hemispheric dome setup for naturalistic visual stimulation in head-fixed mice

Optrode recording of an entorhinal–cortical circuit in freely moving mice

A Head-Mounted Multi-Camera System for Electrophysiology and Behavior in Freely-Moving Mice

Vision Enhanced Neuro-Cognitive Structure for Robotic Spatial Cognition

Brain-wide neural recordings in mice navigating physical spaces enabled by robotic neural recording headstages

Head-mounted Optical Imaging and Optogenetic Stimulation System for Use in Behaving Primates

Combining Unity with machine vision to create low latency, flexible and simple virtual realities

Automatic mapping of multiplexed social receptive fields by deep learning and GPU-accelerated 3D videography

Microprism-based two-photon imaging of the lateral cortex of the mouse inferior colliculus reveals novel organizational principles of the auditory midbrain

Meso-Py: Dual Brain Cortical Calcium Imaging in Mice during Head-Fixed Social Stimulus Presentation

MouseVenue3D: A Markerless Three-Dimension Behavioral Tracking System for Matching Two-Photon Brain Imaging in Free-Moving Mice

A primary sensory cortical interareal feedforward inhibitory circuit for tacto-visual integration

A Platform for Brain-wide Volumetric Functional Ultrasound Imaging and Analysis of Circuit Dynamics in Awake Mice

A mouse model of higher visual cortical function

Hippocampal-dependent navigation in head-fixed mice using a floating real-world environment

A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex

A large-scale, standardized physiological survey reveals higher order coding throughout the mouse visual cortex