DO3D: Self-supervised Learning of Decomposed Object-aware 3D Motion and Depth from Monocular Videos
Xiuzhe Wu,Xiaoyang Lyu,Qihao Huang,Yong Liu,Yang Wu,Ying Shan,Xiaojuan Qi
2024-03-09
Abstract:Although considerable advancements have been attained in self-supervised
depth estimation from monocular videos, most existing methods often treat all
objects in a video as static entities, which however violates the dynamic
nature of real-world scenes and fails to model the geometry and motion of
moving objects. In this paper, we propose a self-supervised method to jointly
learn 3D motion and depth from monocular videos. Our system contains a depth
estimation module to predict depth, and a new decomposed object-wise 3D motion
(DO3D) estimation module to predict ego-motion and 3D object motion. Depth and
motion networks work collaboratively to faithfully model the geometry and
dynamics of real-world scenes, which, in turn, benefits both depth and 3D
motion estimation. Their predictions are further combined to synthesize a novel
video frame for self-supervised training. As a core component of our framework,
DO3D is a new motion disentanglement module that learns to predict camera
ego-motion and instance-aware 3D object motion separately. To alleviate the
difficulties in estimating non-rigid 3D object motions, they are decomposed to
object-wise 6-DoF global transformations and a pixel-wise local 3D motion
deformation field. Qualitative and quantitative experiments are conducted on
three benchmark datasets, including KITTI, Cityscapes, and VKITTI2, where our
model delivers superior performance in all evaluated settings. For the depth
estimation task, our model outperforms all compared research works in the
high-resolution setting, attaining an absolute relative depth error (abs rel)
of 0.099 on the KITTI benchmark. Besides, our optical flow estimation results
(an overall EPE of 7.09 on KITTI) also surpass state-of-the-art methods and
largely improve the estimation of dynamic regions, demonstrating the
effectiveness of our motion model. Our code will be available.
Computer Vision and Pattern Recognition