Abstract:While humans effortlessly discern intrinsic dynamics and adapt to new scenarios, modern AI systems often struggle. Current methods for visual grounding of dynamics either use pure neural-network-based simulators (black box), which may violate physical laws, or traditional physical simulators (white box), which rely on expert-defined equations that may not fully capture actual dynamics. We propose the Neural Material Adaptor (NeuMA), which integrates existing physical laws with learned corrections, facilitating accurate learning of actual dynamics while maintaining the generalizability and interpretability of physical priors. Additionally, we propose Particle-GS, a particle-driven 3D Gaussian Splatting variant that bridges simulation and observed images, allowing back-propagate image gradients to optimize the simulator. Comprehensive experiments on various dynamics in terms of grounded particle accuracy, dynamic rendering quality, and generalization ability demonstrate that NeuMA can accurately capture intrinsic dynamics.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to accurately infer the intrinsic dynamic characteristics of objects from visual observations. Specifically, existing methods have limitations when dealing with visual grounding of dynamics: 1. **Black Box Approaches**: These methods directly use neural networks to simulate dynamic transformations. However, due to the deep coupling between extrinsic properties (such as geometric shapes) and intrinsic physical motions during the rendering process, they are prone to violate physical laws and have limited generalization ability without physical constraints. 2. **White Box Approaches**: These methods rely on traditional physical simulators (such as the material point method) to explicitly approximate the dynamics of objects through partial differential equations (PDEs). However, these methods rely on expert - defined equations and may not be able to fully capture the actual dynamic behavior. To solve these problems, the paper proposes the **Neural Material Adaptor (NeuMA)**, which combines existing physical laws with learned correction terms, thereby achieving accurate learning of actual dynamics while maintaining the generality and interpretability of physical priors. In addition, the paper also proposes a particle - driven 3D Gaussian scattering variant **Particle - GS** to bridge simulated and observed images, allowing the back - propagation of image gradients to optimize the simulator. ### Specific Problem Description - **How to accurately infer the intrinsic dynamic characteristics of objects from visual observations?** - Existing black - box and white - box methods each have limitations and cannot simultaneously ensure accuracy, generalization ability, and physical consistency. - **How to combine physical priors and data - driven learning methods to improve dynamic simulations?** - NeuMA adjusts the expert - designed physical model \(M_0\) by introducing a learnable residual correction term \(\Delta M_\theta\) to make it better adapt to actual observations. ### Solutions - **Neural Material Adaptor (NeuMA)**: - The core idea is to formulate dynamic learning as a residual adaptation paradigm: \(M := M_0+\Delta M\). - \(M_0\) is the expert - designed physical model, and \(\Delta M\) is a term for correction based on observed images. - This method neither relies solely on \(M_0\) like white - box methods nor ignores any physical priors like black - box methods, but combines the advantages of both. - **Particle - GS**: - A differentiable renderer that links simulation results with observed images through a particle - driven 3D Gaussian scattering variant, allowing the simulator to be optimized through image gradients. ### Experimental Verification The paper verifies the effectiveness of NeuMA through multiple experiments, including dynamic scenes with different materials and initial conditions, demonstrating its competitiveness and generalization ability in object dynamic grounding and dynamic scene rendering. In summary, this paper aims to solve the limitations of existing methods in visual grounding of dynamics by combining physical priors and data - driven learning methods, thereby achieving more accurate and more generalized dynamic simulations.

Neural Material Adaptor for Visual Grounding of Intrinsic Dynamics

DEL: Discrete Element Learner for Learning 3D Particle Dynamics with Neural Rendering

NeuroFluid: Fluid Dynamics Grounding with Particle-Driven Neural Radiance Fields

Adaptive Smoothing Gradient Learning for Spiking Neural Networks.

3D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenes

A Neural Material Point Method for Particle-based Simulations

Neural Material: Learning Elastic Constitutive Material and Damping Models from Sparse Data

Subspace Graph Physics: Real-Time Rigid Body-Driven Granular Flow Simulation

DynMF: Neural Motion Factorization for Real-time Dynamic View Synthesis with 3D Gaussian Splatting

PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics

AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulation

Grounding Graph Network Simulators using Physical Sensor Observations

PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification

Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling

Automated 3D Physical Simulation of Open-world Scene with Gaussian Splatting

GNS: A generalizable Graph Neural Network-based simulator for particulate and fluid modeling

Learning to Represent Mechanics via Long-term Extrapolation and Interpolation

MaGS: Reconstructing and Simulating Dynamic 3D Objects with Mesh-adsorbed Gaussian Splatting

Subspace graph networks for real-time granular flow simulation with applications to machine-terrain interactions

Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulation

Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusion