MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian Splatting

Ruijie Zhu,Yanzhe Liang,Hanzhi Chang,Jiacheng Deng,Jiahao Lu,Wenfei Yang,Tianzhu Zhang,Yongdong Zhang

2024-10-10

Abstract:Dynamic scene reconstruction is a long-term challenge in the field of 3D vision. Recently, the emergence of 3D Gaussian Splatting has provided new insights into this problem. Although subsequent efforts rapidly extend static 3D Gaussian to dynamic scenes, they often lack explicit constraints on object motion, leading to optimization difficulties and performance degradation. To address the above issues, we propose a novel deformable 3D Gaussian splatting framework called MotionGS, which explores explicit motion priors to guide the deformation of 3D Gaussians. Specifically, we first introduce an optical flow decoupling module that decouples optical flow into camera flow and motion flow, corresponding to camera movement and object motion respectively. Then the motion flow can effectively constrain the deformation of 3D Gaussians, thus simulating the motion of dynamic objects. Additionally, a camera pose refinement module is proposed to alternately optimize 3D Gaussians and camera poses, mitigating the impact of inaccurate camera poses. Extensive experiments in the monocular dynamic scenes validate that MotionGS surpasses state-of-the-art methods and exhibits significant superiority in both qualitative and quantitative results. Project page: <a class="link-external link-https" href="https://ruijiezhu94.github.io/MotionGS_page" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition,Graphics,Machine Learning

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to solve the key challenges in dynamic scene reconstruction, especially in the field of 3D vision. Specifically, the paper focuses on how to effectively deform 3D Gaussian point clouds (3D Gaussian Splatting) in dynamic scenes to achieve high - quality dynamic scene reconstruction and real - time rendering.

#### Main problems:
1. **Lack of explicit motion constraints**: Although existing methods can extend static 3D Gaussians to dynamic scenes, they often lack explicit constraints on object motion, leading to optimization difficulties and performance degradation.
2. **The impact of inaccurate camera poses**: In dynamic scenes, the estimation of camera poses is usually not accurate enough, which will affect the optimization of 3D Gaussian point clouds and the final rendering quality.

#### Solutions:
To solve the above problems, the author proposes a new deformable 3D Gaussian framework - **MotionGS**. This framework improves the deformation of 3D Gaussian point clouds by introducing explicit motion guidance and is implemented through the following two modules:

1. **Optical Flow Decoupling Module**:
- Decompose 2D optical flow into camera flow and motion flow, where motion flow is only related to object motion, thus providing more direct motion supervision for the deformation of 3D Gaussian point clouds.
- Use motion flow to constrain the deformation of 3D Gaussian point clouds and simulate the motion of dynamic objects.

2. **Camera Pose Refinement Module**:
- Propose an alternating optimization method for 3D Gaussian point clouds and camera poses. Optimize camera poses by back - propagating gradients through photometric consistency loss, reducing the dependence on accurate camera poses and further improving the rendering quality.

#### Main contributions:
- Propose the MotionGS framework, which provides explicit motion guidance and achieves high - quality dynamic scene reconstruction and real - time rendering.
- The Optical Flow Decoupling Module effectively separates the optical flow caused only by object motion, thus efficiently supervising the deformation of 3D Gaussian point clouds.
- The Camera Pose Refinement Module alternately optimizes 3D Gaussian point clouds and camera poses, reducing the dependence on accurate camera poses and further improving the rendering quality.

Through these innovations, the experimental results of MotionGS on multiple datasets show that it significantly outperforms the existing state - of - the - art methods and performs well in both qualitative and quantitative evaluations.