Gaussian Splatting on the Move: Blur and Rolling Shutter Compensation for Natural Camera Motion

Otto Seiskari,Jerry Ylilammi,Valtteri Kaatrasalo,Pekka Rantalankila,Matias Turkulainen,Juho Kannala,Esa Rahtu,Arno Solin

2024-07-17

Abstract:High-quality scene reconstruction and novel view synthesis based on Gaussian Splatting (3DGS) typically require steady, high-quality photographs, often impractical to capture with handheld cameras. We present a method that adapts to camera motion and allows high-quality scene reconstruction with handheld video data suffering from motion blur and rolling shutter distortion. Our approach is based on detailed modelling of the physical image formation process and utilizes velocities estimated using visual-inertial odometry (VIO). Camera poses are considered non-static during the exposure time of a single image frame and camera poses are further optimized in the reconstruction process. We formulate a differentiable rendering pipeline that leverages screen space approximation to efficiently incorporate rolling-shutter and motion blur effects into the 3DGS framework. Our results with both synthetic and real data demonstrate superior performance in mitigating camera motion over existing methods, thereby advancing 3DGS in naturalistic settings.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the impact of image blurring and rolling - shutter distortion caused by camera motion on high - quality scene reconstruction and novel - view synthesis when shooting videos with handheld devices (such as smart phones). Specifically, traditional high - precision scene reconstruction methods (such as 3DGS based on Gaussian point - drawing) usually require stable, high - quality photos, which are often difficult to obtain through handheld devices. The data captured by handheld devices is often affected by motion blur and rolling - shutter distortion, which will significantly reduce the reconstruction quality and increase the possibility of pose - registration failure.

The paper proposes a new method. By modeling the physical imaging process in detail and using the velocity estimated by visual - inertial odometry (VIO) to adapt to camera motion, it allows the use of handheld video data affected by motion blur and rolling - shutter distortion for high - quality scene reconstruction. This method takes into account the non - static characteristics of camera pose within the exposure time of a single image frame and further optimizes the camera pose during the reconstruction process. In addition, the author also designs a differentiable rendering pipeline, which efficiently integrates the rolling - shutter and motion - blur effects into the 3DGS framework using screen - space approximation. Experimental results show that this method is superior to existing methods in alleviating the impact of camera motion, especially in 3DGS applications in natural environments.