Abstract:To study the behavior of freely moving model organisms such as zebrafish (Danio rerio) and fruit flies (Drosophila) across multiple spatial scales, it would be ideal to use a light microscope that can resolve 3D information over a wide field of view (FOV) at high speed and high spatial resolution. However, it is challenging to design an optical instrument to achieve all of these properties simultaneously. Existing techniques for large-FOV microscopic imaging and for 3D image measurement typically require many sequential image snapshots, thus compromising speed and throughput. Here, we present 3D-RAPID, a computational microscope based on a synchronized array of 54 cameras that can capture high-speed 3D topographic videos over a 135-cm^2 area, achieving up to 230 frames per second at throughputs exceeding 5 gigapixels (GPs) per second. 3D-RAPID features a 3D reconstruction algorithm that, for each synchronized temporal snapshot, simultaneously fuses all 54 images seamlessly into a globally-consistent composite that includes a coregistered 3D height map. The self-supervised 3D reconstruction algorithm itself trains a spatiotemporally-compressed convolutional neural network (CNN) that maps raw photometric images to 3D topography, using stereo overlap redundancy and ray-propagation physics as the only supervision mechanism. As a result, our end-to-end 3D reconstruction algorithm is robust to generalization errors and scales to arbitrarily long videos from arbitrarily sized camera arrays. The scalable hardware and software design of 3D-RAPID addresses a longstanding problem in the field of behavioral imaging, enabling parallelized 3D observation of large collections of freely moving organisms at high spatiotemporal throughputs, which we demonstrate in ants (Pogonomyrmex barbatus), fruit flies, and zebrafish larvae.

Gigapixel imaging with a novel multi-camera array microscope

Video-rate Imaging of Biological Dynamics at Centimetre Scale and Micrometre Resolution

Calibration of a Camera-Array-based Microscopic System with Spatiotemporal Structured Light Encoding

Parallelized computational 3D video microscopy of freely moving organisms at multiple gigapixels per second

Real-time multi-angle projection imaging of biological dynamics

Pan-cortical cellular imaging in freely behaving mice using a miniaturized micro-camera array microscope (mini-MCAM)

High-speed 3D Imaging with 25-Camera Multifocus Microscope

Video-rate large-scale imaging with Multi-Z confocal microscopy

A three-camera imaging microscope for high-speed single-molecule tracking and super-resolution imaging in living cells

Computational 3D topographic microscopy from terabytes of data per sample

A miniature fluorescence microscope for multi-plane imaging

Learning Consistent Subcellular Landmarks to Quantify Changes in Multiplexed Protein Maps.

Rapid Image Deconvolution and Multiview Fusion for Optical Microscopy.

Simultaneous Multiview Capture and Fusion Improves Spatial Resolution in Wide-Field and Light-Sheet Microscopy

Arthropod Eye-Inspired Digital Camera With Unique Imaging Characteristics

Expansion-assisted selective plane illumination microscopy for nanoscale imaging of centimeter-scale tissues

Multiscale Light Field Microscopy Platform for Multi-purpose Dynamic Volumetric Bioimaging

Single-Shot 3D Widefield Fluorescence Imaging with a Computational Miniature Mesoscope

Imaging of cellular dynamics and : from a whole organism to sub-cellular imaging with self-driving, multi-scale microscopy

An integrated platform for high-throughput nanoscopy

Volumetric trans-scale imaging of massive quantity of heterogeneous cell populations in centimeter-wide tissue and embryo