Abstract:Reconstructing 4D scenes from video inputs is a crucial yet challenging task. Conventional methods usually rely on the assumptions of multi-view video inputs, known camera parameters, or static scenes, all of which are typically absent under in-the-wild scenarios. In this paper, we relax all these constraints and tackle a highly ambitious but practical task, which we termed as AnyV4D: we assume only one monocular video is available without any camera parameters as input, and we aim to recover the dynamic 4D world alongside the camera poses. To this end, we introduce GFlow, a new framework that utilizes only 2D priors (depth and optical flow) to lift a video (3D) to a 4D explicit representation, entailing a flow of Gaussian splatting through space and time. GFlow first clusters the scene into still and moving parts, then applies a sequential optimization process that optimizes camera poses and the dynamics of 3D Gaussian points based on 2D priors and scene clustering, ensuring fidelity among neighboring points and smooth movement across frames. Since dynamic scenes always introduce new content, we also propose a new pixel-wise densification strategy for Gaussian points to integrate new visual content. Moreover, GFlow transcends the boundaries of mere 4D reconstruction; it also enables tracking of any points across frames without the need for prior training and segments moving objects from the scene in an unsupervised way. Additionally, the camera poses of each frame can be derived from GFlow, allowing for rendering novel views of a video scene through changing camera pose. By employing the explicit representation, we may readily conduct scene-level or object-level editing as desired, underscoring its versatility and power. Visit our project website at: <a class="link-external link-https" href="https://littlepure2333.github.io/GFlow" rel="external noopener nofollow">this https URL</a>

A New Non-Linear Algorithm for 3D Reconstruction from Straight-line Optical Flow

Linear Model for 3d Motion Estimation and Shape Reconstruction Based on the Straight-Line Optical Flow

Unsupervised Learning of Scene Flow Estimation Fusing with Local Rigidity.

Three-dimensional motion estimation by straight-line and endpoint optical flow

A Velocity Decomposition-Based 3D Optical Flow Method for Accurate Tomo-PIV Measurement

A Guided Filter-Based 3D Hybrid Variational Optical Flow for Accurate Tomographic PIV Measurements

Recurrent Graph Optimal Transport for Learning 3D Flow Motion in Particle Tracking

Direct Method for Motion Estimation and Structure Reconstruction Based on Optical Flow

A NONLINEAR FILTERING BASED OPTICAL FLOW COMPUTATION

A Variational Method for Recovering 3D Structure of Purely Rotational Object from Image Sequence

A Robust Optical Flow Computation

Optical Flow in Mostly Rigid Scenes

Accurate dense optical flow estimation using adaptive structure tensors and a parametric model

Dynamic 3d Scene Depth Reconstruction Via Optical Flow Field Rectification

3D Facial Geometry Recovery Via Group-Wise Optical Flow

Robust Optical Flow Computation: A Higher-Order Differential Approach

Dynamic Shape Capture Via Periodical-Illumination Optical Flow Estimation and Multi-view Photometric Stereo.

Joint direct estimation of 3D geometry and 3D motion using spatio temporal gradients

Motion and Structure from Event-based Normal Flow

Optical Flow Estimation Combining Spatial-Temporal Derivatives Based Nonlinear Filtering.

GFlow: Recovering 4D World from Monocular Video