Abstract:While conventional cameras offer versatility for applications ranging from amateur photography to autonomous driving, computational cameras allow for domain-specific adaption. Cameras with co-designed optics and image processing algorithms enable high-dynamic-range image recovery, depth estimation, and hyperspectral imaging through optically encoding scene information that is otherwise undetected by conventional cameras. However, this optical encoding creates a challenging inverse reconstruction problem for conventional image recovery, and often lowers the overall photographic quality. Thus computational cameras with domain-specific optics have only been adopted in a few specialized applications where the captured information cannot be acquired in other ways. In this work, we investigate a method that combines two optical systems into one to tackle this challenge. We split the aperture of a conventional camera into two halves: one which applies an application-specific modulation to the incident light via a diffractive optical element to produce a coded image capture, and one which applies no modulation to produce a conventional image capture. Co-designing the phase modulation of the split aperture with a dual-pixel sensor allows us to simultaneously capture these coded and uncoded images without increasing physical or computational footprint. With an uncoded conventional image alongside the optically coded image in hand, we investigate image reconstruction methods that are conditioned on the conventional image, making it possible to eliminate artifacts and compute costs that existing methods struggle with. We assess the proposed method with 2-in-1 cameras for optical high-dynamic-range reconstruction, monocular depth estimation, and hyperspectral imaging, comparing favorably to all tested methods in all applications.

Computational photography with novel camera sensors

Fast Image Super-Resolution for a Dual-Resolution Camera

Continuous Digital Zoom with Cross Attention for Dual Camera System

Photo-realistic continuous digital zooming for an asymmetrical dual camera system

Superpixel-Based Linear Reconstruction Method for Dual-Camera Compressed Hyperspectral Imaging System

Computational Photography：Keys, Methods and Applications

Split-Aperture 2-in-1 Computational Cameras

Extreme-Quality Computational Imaging via Degradation Framework

Recent Advances in Computational Photography

An Overview of Computational Photography

HDR Imaging with Quanta Image Sensors: Theoretical Limits and Optimal Reconstruction

All-day Thin-Lens Computational Imaging with Scene-Specific Learning Recovery

Noise and dynamic range optimal computational imaging

Camera Lens Super-Resolution

Computational Hyperspectral Imaging

A Dual Sensor Computational Camera for High Quality Dark Videography

Low-Light Color Imaging Via Cross-Camera Synthesis.

A Mutually Boosting Dual Sensor Computational Camera for High Quality Dark Videography.

A Virtual-Sensor Construction Network Based on Physical Imaging for Image Super-Resolution

End-to-end Learned, Optically Coded Super-resolution SPAD Camera

A novel super-resolution imaging method based on TDI CCD charge transfer and random exposure