Abstract:To study the behavior of freely moving model organisms such as zebrafish (Danio rerio) and fruit flies (Drosophila) across multiple spatial scales, it would be ideal to use a light microscope that can resolve 3D information over a wide field of view (FOV) at high speed and high spatial resolution. However, it is challenging to design an optical instrument to achieve all of these properties simultaneously. Existing techniques for large-FOV microscopic imaging and for 3D image measurement typically require many sequential image snapshots, thus compromising speed and throughput. Here, we present 3D-RAPID, a computational microscope based on a synchronized array of 54 cameras that can capture high-speed 3D topographic videos over a 135-cm^2 area, achieving up to 230 frames per second at throughputs exceeding 5 gigapixels (GPs) per second. 3D-RAPID features a 3D reconstruction algorithm that, for each synchronized temporal snapshot, simultaneously fuses all 54 images seamlessly into a globally-consistent composite that includes a coregistered 3D height map. The self-supervised 3D reconstruction algorithm itself trains a spatiotemporally-compressed convolutional neural network (CNN) that maps raw photometric images to 3D topography, using stereo overlap redundancy and ray-propagation physics as the only supervision mechanism. As a result, our end-to-end 3D reconstruction algorithm is robust to generalization errors and scales to arbitrarily long videos from arbitrarily sized camera arrays. The scalable hardware and software design of 3D-RAPID addresses a longstanding problem in the field of behavioral imaging, enabling parallelized 3D observation of large collections of freely moving organisms at high spatiotemporal throughputs, which we demonstrate in ants (Pogonomyrmex barbatus), fruit flies, and zebrafish larvae.

Line excitation array detection fluorescence microscopy at 0.8 million frames per second

Dynamic live-cell super-resolution imaging with parallelized fluorescence emission difference microscopy

Real-Time Super-Resolution Imaging By High-Speed Fluorescence Emission Difference Microscopy

Parallelized Fluorescence Lifetime Imaging Microscopy (FLIM) Based on Photon Reassignment

Digitally synthesized beat frequency multiplexing for sub-millisecond fluorescence microscopy

High-speed two-photon microscopy with adaptive line-excitation

Spectro-temporal encoded Multiphoton Microscopy

High-speed optical imaging with sCMOS pixel reassignment

Ultra-fast Line Scan Microscopic Imaging System

Pixel multiplexing for high-speed multi-resolution fluorescence imaging

Pixel-wise programmability enables dynamic high-SNR cameras for high-speed microscopy

Analog multiplexing of laser clock and computational photon counting for fast fluorescence lifetime imaging microscopy

Resonant Electro-optic Imaging for Microscopy at Nanosecond Resolution

Efficient wide-field FLIM

Serial time-encoded amplified imaging for real-time observation of fast dynamic phenomena

Kilohertz frame-rate two-photon tomography

A three-camera imaging microscope for high-speed single-molecule tracking and super-resolution imaging in living cells

Parallelized computational 3D video microscopy of freely moving organisms at multiple gigapixels per second

Imaging Flow Cytometry Using Linear Array Spot Excitation

Ten-kilohertz two-photon microscopy imaging of single-cell dendritic activity and hemodynamics in vivo

Multi-channel beam-scanning imaging at kHz frame rates by Lissajous trajectory microscopy