Abstract:Monocular dynamic reconstruction is a challenging and long-standing vision problem due to the highly ill-posed nature of the task. Existing approaches are limited in that they either depend on templates, are effective only in quasi-static scenes, or fail to model 3D motion explicitly. In this work, we introduce a method capable of reconstructing generic dynamic scenes, featuring explicit, full-sequence-long 3D motion, from casually captured monocular videos. We tackle the under-constrained nature of the problem with two key insights: First, we exploit the low-dimensional structure of 3D motion by representing scene motion with a compact set of SE3 motion bases. Each point's motion is expressed as a linear combination of these bases, facilitating soft decomposition of the scene into multiple rigidly-moving groups. Second, we utilize a comprehensive set of data-driven priors, including monocular depth maps and long-range 2D tracks, and devise a method to effectively consolidate these noisy supervisory signals, resulting in a globally consistent representation of the dynamic scene. Experiments show that our method achieves state-of-the-art performance for both long-range 3D/2D motion estimation and novel view synthesis on dynamic scenes. Project Page: <a class="link-external link-https" href="https://shape-of-motion.github.io/" rel="external noopener nofollow">this https URL</a>

3-D shape recognition by active vision-without camera velocity information

Moving Object Tracking Research Based on Active Vision

Motion Detection for Rapidly Moving Cameras in Fully 3D Scenes

Active and direct acquisition of 3D map in robot by combining motion and perceived images

Linear Model for 3d Motion Estimation and Shape Reconstruction Based on the Straight-Line Optical Flow

3D Space Recognition of Robot by Combining Action and Visual Perception

A theory on the determination of 3D motion and 3D structure from features

Shape of Motion: 4D Reconstruction from a Single Video

Dynamic Shape Capture Via Periodical-Illumination Optical Flow Estimation and Multi-view Photometric Stereo.

Visual Odometry in dynamical scenes

Motion Recovery Using the Image Interpolation Algorithm and an RGB-D Camera

Real-time non-rigid shape recovery via active appearance models for augmented reality

3D Target Detection and Tracking Based on Scene Flow

Unsupervised Learning of Scene Flow Estimation Fusing with Local Rigidity.

3-D rigid body tracking using vision and depth sensors

Three-dimensional motion estimation by straight-line and endpoint optical flow

Detecting Moving Objects Using a Novel Optical-Flow-Based Range-Independent Invariant

Active Acquisition of 3D Map in Robot Brain by Combining Motion and Perceived Images

Direct Method for Motion Estimation and Structure Reconstruction Based on Optical Flow

3D Video Based Segmentation and Motion Estimation with Active Surface Evolution

A Variational Method for Recovering 3D Structure of Purely Rotational Object from Image Sequence