R4D-planes: Remapping Planes for Novel View Synthesis and Self-Supervised Decoupling of Monocular Videos

Junyuan Guo,Hao Tang,Teng Wang,Chao Wang

DOI: https://doi.org/10.1145/3664647.3681281

2024-01-01

Abstract:The tasks of view synthesis and decoupling dynamic objects from the static environment for monocular scenes are both long-standing challenges in CV and CG. Most of the previous NeRF-based methods rely on implicit representation, which require additional supervision and training time. Later, various explicit representations like multi-planes or 3D gaussian splatting have been extended and applied to the task of novel view synthesis for dynamic scenes. They introduce an additional time dimension or a deformation field into the original representation to encode dynamics. Due to the effective explicit representations, these methods greatly reduce the time consumption, but still fail to achieve high rendering quality in some scenes, especially for some real scenes. For the latter decoupling problem, previous neural radiation field methods require frequent tuning of the relevant parameters for different scenes, which is very inconvenient for practical use. We consider above problems and propose a new representation of dynamic scenes based on tensor decomposition, which we call R4D-planes. The key to our method is remapping, which compensates for the shortcomings of the plane structure by fusing space-time information and remapping to new indexes. Furthermore, we implement a new decoupling structure, which can efficiently decouple dynamic and static scenes in a self-supervised manner. Experimental results show our method achieves better rendering quality and training efficiency in both view synthesis and decoupling tasks for monocular scenes.