Abstract:The ability to present three-dimensional (3D) scenes with continuous depth sensation has a profound impact on virtual and augmented reality, human–computer interaction, education and training. Computer-generated holography (CGH) enables high-spatio-angular-resolution 3D projection via numerical simulation of diffraction and interference<a href="/articles/s41586-020-03152-0#ref-CR1">1</a>. Yet, existing physically based methods fail to produce holograms with both per-pixel focal control and accurate occlusion<a href="/articles/s41586-020-03152-0#ref-CR2">2</a>,<a href="/articles/s41586-020-03152-0#ref-CR3">3</a>. The computationally taxing Fresnel diffraction simulation further places an explicit trade-off between image quality and runtime, making dynamic holography impractical<a href="/articles/s41586-020-03152-0#ref-CR4">4</a>. Here we demonstrate a deep-learning-based CGH pipeline capable of synthesizing a photorealistic colour 3D hologram from a single RGB-depth image in real time. Our convolutional neural network (CNN) is extremely memory efficient (below 620 kilobytes) and runs at 60 hertz for a resolution of 1,920 × 1,080 pixels on a single consumer-grade graphics processing unit. Leveraging low-power on-device artificial intelligence acceleration chips, our CNN also runs interactively on mobile (iPhone 11 Pro at 1.1 hertz) and edge (Google Edge TPU at 2.0 hertz) devices, promising real-time performance in future-generation virtual and augmented-reality mobile headsets. We enable this pipeline by introducing a large-scale CGH dataset (MIT-CGH-4K) with 4,000 pairs of RGB-depth images and corresponding 3D holograms. Our CNN is trained with differentiable wave-based loss functions<a href="/articles/s41586-020-03152-0#ref-CR5">5</a> and physically approximates Fresnel diffraction. With an anti-aliasing phase-only encoding method, we experimentally demonstrate speckle-free, natural-looking, high-resolution 3D holograms. Our learning-based approach and the Fresnel hologram dataset will help to unlock the full potential of holography and enable applications in metasurface design<a href="/articles/s41586-020-03152-0#ref-CR6">6</a>,<a href="/articles/s41586-020-03152-0#ref-CR7">7</a>, optical and acoustic tweezer-based microscopic manipulation<a href="#ref-CR8">8</a>,<a href="#ref-CR9">9</a>,<a href="/articles/s41586-020-03152-0#ref-CR10">10</a>, holographic microscopy<a href="/articles/s41586-020-03152-0#ref-CR11">11</a> and single-exposure volumetric 3D printing<a href="/articles/s41586-020-03152-0#ref-CR12">12</a>,<a href="/articles/s41586-020-03152-0#ref-CR13">13</a>.

Deep Neural Network-Based Phase-Recovery and Auto-Focusing Extend the Depth-of-Field in Digital Holography

Extended Depth-of-field in Holographic Imaging Using Deep-Learning-based Autofocusing and Phase Recovery

Auto-focusing and extended depth-of-field holographic reconstruction using deep learning (Conference Presentation)

Holographic image reconstruction with phase recovery and autofocusing using recurrent neural networks

Fast phase retrieval in off-axis digital holographic microscopy through deep learning.

eFIN: Enhanced Fourier Imager Network for Generalizable Autofocusing and Pixel Super-Resolution in Holographic Imaging

Generating Multi‐Depth 3D Holograms Using a Fully Convolutional Neural Network

Phase Retrieval Combined with the Deep Learning Denoising Method in Holographic Data Storage

Automated phase reconstruction and super-resolution with deep learning in digital holography

Computer Holography Using Deep Neural Network with Fourier Basis.

Generating Real-Scene Hologram Through Light Field Imaging and Deep Learning

Denoising Method Based on Deep Learning Used in Phase Retrieval of Holographic Storage

54.2: High‐fidelity model‐driven deep learning network for phase‐only computer‐generated holography

Lensless Phase Retrieval Based on Deep Learning Used in Holographic Data Storage

Extension of Refocus Depth Range in Digital Holographic Microscopy.

Towards real-time photorealistic 3D holography with deep neural networks

OAH-Net: A Deep Neural Network for Hologram Reconstruction of Off-axis Digital Holographic Microscope

High-speed Computer-Generated Holography Using an Autoencoder-Based Deep Neural Network

Deep-learning-aided multi-focal hologram generation

Numerical phase-detection autofocusing method for digital holography reconstruction processing

High-fidelity, Model-Driven Deep Learning Network for Phase-Only Computer-Generated Holography (conference Presentation)