Abstract:3D reconstruction and simulation, although interrelated, have distinct objectives: reconstruction requires a flexible 3D representation that can adapt to diverse scenes, while simulation needs a structured representation to model motion principles effectively. This paper introduces the Mesh-adsorbed Gaussian Splatting (MaGS) method to address this challenge. MaGS constrains 3D Gaussians to roam near the mesh, creating a mutually adsorbed mesh-Gaussian 3D representation. Such representation harnesses both the rendering flexibility of 3D Gaussians and the structured property of meshes. To achieve this, we introduce RMD-Net, a network that learns motion priors from video data to refine mesh deformations, alongside RGD-Net, which models the relative displacement between the mesh and Gaussians to enhance rendering fidelity under mesh constraints. To generalize to novel, user-defined deformations beyond input video without reliance on temporal data, we propose MPE-Net, which leverages inherent mesh information to bootstrap RMD-Net and RGD-Net. Due to the universality of meshes, MaGS is compatible with various deformation priors such as ARAP, SMPL, and soft physics simulation. Extensive experiments on the D-NeRF, DG-Mesh, and PeopleSnapshot datasets demonstrate that MaGS achieves state-of-the-art performance in both reconstruction and simulation.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **How to reconstruct and simulate dynamic 3D objects from monocular videos, and achieve high - quality rendering and realistic deformation**. Specifically, traditional 3D reconstruction and simulation methods usually handle these two tasks separately. 3D reconstruction requires a flexible 3D representation to adapt to different scenarios, while simulation requires a structured representation to effectively model the motion principles. It is difficult to satisfy these two requirements simultaneously in a single framework. To address this challenge, the authors propose the Mesh - adsorbed Gaussian Splatting (MaGS) method. ### Main problems 1. **Unified framework for reconstruction and simulation**: - 3D reconstruction requires a flexible representation method to adapt to various scenarios. - Simulation requires a structured representation method to effectively model the deformation rules. 2. **High - precision reconstruction and realistic simulation of dynamic objects**: - How to ensure that the 3D objects obtained during the reconstruction process can maintain realistic motion and deformation in the simulation. - How to generalize to user - defined new deformations without relying on time information. ### Core contributions of MaGS 1. **Hybrid representation**: - MaGS creates a hybrid representation that combines the rendering flexibility and structured properties by constraining 3D Gaussian distributions near triangular meshes. This representation takes advantage of the rendering flexibility of 3D Gaussian distributions and retains the structural properties of the meshes. 2. **Network design**: - RMD - Net (Relative Mesh Deformation Network) learns the motion priors in video data to optimize mesh deformation. - RGD - Net (Relative Gaussian Deformation Network) models the relative displacements between meshes and Gaussians to improve the rendering fidelity under mesh constraints. - MPE - Net (Mesh Pose Embedding Network) uses the inherent information of meshes to guide relative deformations, enabling MaGS to generalize to new deformations outside the input video. 3. **Joint optimization**: - By jointly optimizing meshes, 3D Gaussian distributions, and network parameters, MaGS achieves high - precision rendering and realistic deformation. ### Experimental results The authors conducted extensive experiments on multiple datasets (such as D - NeRF, DG - Mesh, and PeopleSnapshot) and verified the superior performance of MaGS in reconstruction and simulation tasks. In conclusion, MaGS provides a brand - new paradigm that unifies 3D reconstruction and simulation within a single framework, solving the limitations of traditional methods in these two tasks.

MaGS: Reconstructing and Simulating Dynamic 3D Objects with Mesh-adsorbed Gaussian Splatting

Mesh-based Gaussian Splatting for Real-time Large-scale Deformation

Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Monocular Videos

DyGASR: Dynamic Generalized Exponential Splatting with Surface Alignment for Accelerated 3D Mesh Reconstruction

MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian Splatting

MeshGS: Adaptive Mesh-Aligned Gaussian Splatting for High-Quality Rendering

Mani-GS: Gaussian Splatting Manipulation with Triangular Mesh

MVG-Splatting: Multi-View Guided Gaussian Splatting with Adaptive Quantile-Based Geometric Consistency Densification

GaMeS: Mesh-Based Adapting and Modification of Gaussian Splatting

SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering

Motion-aware 3D Gaussian Splatting for Efficient Dynamic Scene Reconstruction

3D Geometry-aware Deformable Gaussian Splatting for Dynamic View Synthesis

SAGS: Structure-Aware 3D Gaussian Splatting

3D Gaussian Model for Animation and Texturing

Deform3DGS: Flexible Deformation for Fast Surgical Scene Reconstruction with Gaussian Splatting

D-MiSo: Editing Dynamic 3D Scenes using Multi-Gaussians Soup

DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation

SplatFields: Neural Gaussian Splats for Sparse 3D and 4D Reconstruction

Event3DGS: Event-Based 3D Gaussian Splatting for High-Speed Robot Egomotion

DOGS: Distributed-Oriented Gaussian Splatting for Large-Scale 3D Reconstruction Via Gaussian Consensus