Abstract:The ability to present three-dimensional (3D) scenes with continuous depth sensation has a profound impact on virtual and augmented reality, human–computer interaction, education and training. Computer-generated holography (CGH) enables high-spatio-angular-resolution 3D projection via numerical simulation of diffraction and interference<a href="/articles/s41586-020-03152-0#ref-CR1">1</a>. Yet, existing physically based methods fail to produce holograms with both per-pixel focal control and accurate occlusion<a href="/articles/s41586-020-03152-0#ref-CR2">2</a>,<a href="/articles/s41586-020-03152-0#ref-CR3">3</a>. The computationally taxing Fresnel diffraction simulation further places an explicit trade-off between image quality and runtime, making dynamic holography impractical<a href="/articles/s41586-020-03152-0#ref-CR4">4</a>. Here we demonstrate a deep-learning-based CGH pipeline capable of synthesizing a photorealistic colour 3D hologram from a single RGB-depth image in real time. Our convolutional neural network (CNN) is extremely memory efficient (below 620 kilobytes) and runs at 60 hertz for a resolution of 1,920 × 1,080 pixels on a single consumer-grade graphics processing unit. Leveraging low-power on-device artificial intelligence acceleration chips, our CNN also runs interactively on mobile (iPhone 11 Pro at 1.1 hertz) and edge (Google Edge TPU at 2.0 hertz) devices, promising real-time performance in future-generation virtual and augmented-reality mobile headsets. We enable this pipeline by introducing a large-scale CGH dataset (MIT-CGH-4K) with 4,000 pairs of RGB-depth images and corresponding 3D holograms. Our CNN is trained with differentiable wave-based loss functions<a href="/articles/s41586-020-03152-0#ref-CR5">5</a> and physically approximates Fresnel diffraction. With an anti-aliasing phase-only encoding method, we experimentally demonstrate speckle-free, natural-looking, high-resolution 3D holograms. Our learning-based approach and the Fresnel hologram dataset will help to unlock the full potential of holography and enable applications in metasurface design<a href="/articles/s41586-020-03152-0#ref-CR6">6</a>,<a href="/articles/s41586-020-03152-0#ref-CR7">7</a>, optical and acoustic tweezer-based microscopic manipulation<a href="#ref-CR8">8</a>,<a href="#ref-CR9">9</a>,<a href="/articles/s41586-020-03152-0#ref-CR10">10</a>, holographic microscopy<a href="/articles/s41586-020-03152-0#ref-CR11">11</a> and single-exposure volumetric 3D printing<a href="/articles/s41586-020-03152-0#ref-CR12">12</a>,<a href="/articles/s41586-020-03152-0#ref-CR13">13</a>.

High-speed Three-Dimensional Digital Holographic Reconstruction of Particles Based on a YOLO Network

Rapid Algorithm for Computer-Generated Hologram of Three-Dimensional Object

Holographic 3D Particle Reconstruction Using a One-Stage Network

Research on 3D Particle Field Reconstruction Method Based on Deep Learning

Adaptive In-Focus Particle Detection and Segmentation in Holographic 3d Image with Mechanism-Guided Machine Learning

Towards real-time photorealistic 3D holography with deep neural networks

Dense-U-net: Dense Encoder–decoder Network for Holographic Imaging of 3D Particle Fields

OAH-Net: A Deep Neural Network for Hologram Reconstruction of Off-axis Digital Holographic Microscope

Holo-U2Net for High-Fidelity 3D Hologram Generation

High-speed Computer-Generated Holography Using an Autoencoder-Based Deep Neural Network

Reconstruction of partially occluded objects with a physics-driven self-training neural network

Unsupervised deep neural network for fast computer-generated holography with continuous depth control

Generative adversarial neural network for 3D-hologram reconstruction

High-fidelity, Model-Driven Deep Learning Network for Phase-Only Computer-Generated Holography (conference Presentation)

Unsupervised Deep Learning Enables 3D Imaging for Single‐Shot Incoherent Holography

Machine learning holography for 3D particle field imaging

Generalizable Deep Learning Approach for 3D Particle Imaging using Holographic Microscopy

Generating Multi‐Depth 3D Holograms Using a Fully Convolutional Neural Network

Y4-Net: a Deep Learning Solution to One-Shot Dual-Wavelength Digital Holographic Reconstruction.

Unpaired Learning for Digital Holographic Reconstruction and Generation

Neural holography with camera-in-the-loop training