Abstract:3D Gaussian Splatting-based techniques have recently advanced 3D scene reconstruction and novel view synthesis, achieving high-quality real-time rendering. However, these approaches are inherently limited by the underlying pinhole camera assumption in modeling the images and hence only work for All-in-Focus (AiF) sharp image inputs. This severely affects their applicability in real-world scenarios where images often exhibit defocus blur due to the limited depth-of-field (DOF) of imaging devices. Additionally, existing 3D Gaussian Splatting (3DGS) methods also do not support rendering of DOF effects. To address these challenges, we introduce DOF-GS that allows for rendering adjustable DOF effects, removing defocus blur as well as refocusing of 3D scenes, all from multi-view images degraded by defocus blur. To this end, we re-imagine the traditional Gaussian Splatting pipeline by employing a finite aperture camera model coupled with explicit, differentiable defocus rendering guided by the Circle-of-Confusion (CoC). The proposed framework provides for dynamic adjustment of DOF effects by changing the aperture and focal distance of the underlying camera model on-demand. It also enables rendering varying DOF effects of 3D scenes post-optimization, and generating AiF images from defocused training images. Furthermore, we devise a joint optimization strategy to further enhance details in the reconstructed scenes by jointly optimizing rendered defocused and AiF images. Our experimental results indicate that DOF-GS produces high-quality sharp all-in-focus renderings conditioned on inputs compromised by defocus blur, with the training process incurring only a modest increase in GPU memory consumption. We further demonstrate the applications of the proposed method for adjustable defocus rendering and refocusing of the 3D scene from input images degraded by defocus blur.

Accurate Depth-of-Field Rendering Using Adaptive Bilateral Depth Filtering.

Layered depth-of-field rendering using color spreading.

Real-time depth-of-field rendering using single-layer composition

Depth-of-Field Rendering with Saliency-Based Bilateral Filtering

Depth of Field Rendering Using Multilayer-Neighborhood Optimization

Multi-image-distance Imaging System for Extending Depth-of-field

An Asymmetric Edge Adaptive Filter for Depth Generation and Hole Filling in 3DTV

Efficient Depth-Of-Field Rendering with Adaptive Sampling and Multiscale Reconstruction

Calibration-free Deep Optics for Depth Estimation with Precise Simulation

Image-based Depth-of-field Rendering with Non-Local Means Filtering

Real-time depth-of-field rendering for shadow

Hierarchical Joint Bilateral Filtering for Depth Post-Processing

Real-Time Rendering Based on GPU for Binocular Stereo System

A Hybrid System for Real-time Rendering of Depth of Field Effect in Games

Depth-Image-Based Rendering Based on GPU-accelerated

DOF-GS: Adjustable Depth-of-Field 3D Gaussian Splatting for Refocusing,Defocus Rendering and Blur Removal

Multi-channel Residual Network Model for Accurate Estimation of Spatially-Varying and Depth-Dependent Defocus Kernels.

Depth-guided Deep Filtering Network for Efficient Single Image Bokeh Rendering

Predicted Virtual Soft Shadow Maps with High Quality Filtering

A New Virtual View Rendering Method Based on Depth Image

Defocusing Response Function Optimization in Depth Estimation of Boundary Complex Light Field Image