Abstract:The ability to present three-dimensional (3D) scenes with continuous depth sensation has a profound impact on virtual and augmented reality, human–computer interaction, education and training. Computer-generated holography (CGH) enables high-spatio-angular-resolution 3D projection via numerical simulation of diffraction and interference<a href="/articles/s41586-020-03152-0#ref-CR1">1</a>. Yet, existing physically based methods fail to produce holograms with both per-pixel focal control and accurate occlusion<a href="/articles/s41586-020-03152-0#ref-CR2">2</a>,<a href="/articles/s41586-020-03152-0#ref-CR3">3</a>. The computationally taxing Fresnel diffraction simulation further places an explicit trade-off between image quality and runtime, making dynamic holography impractical<a href="/articles/s41586-020-03152-0#ref-CR4">4</a>. Here we demonstrate a deep-learning-based CGH pipeline capable of synthesizing a photorealistic colour 3D hologram from a single RGB-depth image in real time. Our convolutional neural network (CNN) is extremely memory efficient (below 620 kilobytes) and runs at 60 hertz for a resolution of 1,920 × 1,080 pixels on a single consumer-grade graphics processing unit. Leveraging low-power on-device artificial intelligence acceleration chips, our CNN also runs interactively on mobile (iPhone 11 Pro at 1.1 hertz) and edge (Google Edge TPU at 2.0 hertz) devices, promising real-time performance in future-generation virtual and augmented-reality mobile headsets. We enable this pipeline by introducing a large-scale CGH dataset (MIT-CGH-4K) with 4,000 pairs of RGB-depth images and corresponding 3D holograms. Our CNN is trained with differentiable wave-based loss functions<a href="/articles/s41586-020-03152-0#ref-CR5">5</a> and physically approximates Fresnel diffraction. With an anti-aliasing phase-only encoding method, we experimentally demonstrate speckle-free, natural-looking, high-resolution 3D holograms. Our learning-based approach and the Fresnel hologram dataset will help to unlock the full potential of holography and enable applications in metasurface design<a href="/articles/s41586-020-03152-0#ref-CR6">6</a>,<a href="/articles/s41586-020-03152-0#ref-CR7">7</a>, optical and acoustic tweezer-based microscopic manipulation<a href="#ref-CR8">8</a>,<a href="#ref-CR9">9</a>,<a href="/articles/s41586-020-03152-0#ref-CR10">10</a>, holographic microscopy<a href="/articles/s41586-020-03152-0#ref-CR11">11</a> and single-exposure volumetric 3D printing<a href="/articles/s41586-020-03152-0#ref-CR12">12</a>,<a href="/articles/s41586-020-03152-0#ref-CR13">13</a>.

Research on 3D Particle Field Reconstruction Method Based on Deep Learning

A Calibration-Informed Deep Learning Model for Three-Dimensional Particle Reconstruction of Volumetric Particle Image Velocimetry

Digital Holography of Particle Fields: Reconstruction by Use of Complex Amplitude.

Machine learning holography for 3D particle field imaging

Dense-U-net: Dense Encoder–decoder Network for Holographic Imaging of 3D Particle Fields

High-speed Three-Dimensional Digital Holographic Reconstruction of Particles Based on a YOLO Network

AI-assisted Reconstruction of Cosmic Velocity Field from Redshift-Space Spatial Distribution of Halos

Adaptive In-Focus Particle Detection and Segmentation in Holographic 3d Image with Mechanism-Guided Machine Learning

Generating Real-Scene Hologram Through Light Field Imaging and Deep Learning

Characterization Method for Particle Extraction from Raw-Reconstructed Images Using U-Net

Generalizable Deep Learning Approach for 3D Particle Imaging using Holographic Microscopy

B-U-net: Holographic image segmentation of multi-scale dense particle field with noisy training dataset

Reconstruction of Particle Flow Energy Distribution Using Deep Learning Algorithms

UHRNet: A Deep Learning-Based Method for Accurate 3D Reconstruction from a Single Fringe-Pattern

Real-Time 3D Tracking of Multi-Particle in the Wide-Field Illumination Based on Deep Learning

Unsupervised Deep Learning Enables 3D Imaging for Single‐Shot Incoherent Holography

Reconstruction of partially occluded objects with a physics-driven self-training neural network

Generating Multi‐Depth 3D Holograms Using a Fully Convolutional Neural Network

Computer Holography Using Deep Neural Network with Fourier Basis.

Towards real-time photorealistic 3D holography with deep neural networks

Learning Reliable Gradients from Undersampled Circular Light Field for 3D Reconstruction.