Abstract:A physical simulation engine (PSE) is a software system that simulates physical environments and objects. Modern PSEs feature both forward and backward simulations, where the forward phase predicts the behavior of a simulated system, and the backward phase provides gradients (guidance) for learning-based control tasks, such as a robot arm learning to fetch items. This way, modern PSEs show promising support for learning-based control methods. To date, PSEs have been largely used in various high-profitable, commercial applications, such as games, movies, virtual reality (VR), and robotics. Despite the prosperous development and usage of PSEs by academia and industrial manufacturers such as Google and NVIDIA, PSEs may produce incorrect simulations, which may lead to negative results, from poor user experience in entertainment to accidents in robotics-involved manufacturing and surgical operations.
This paper introduces PHYFU, a fuzzing framework designed specifically for PSEs to uncover errors in both forward and backward simulation phases. PHYFU mutates initial states and asserts if the PSE under test behaves consistently with respect to basic Physics Laws (PLs). We further use feedback-driven test input scheduling to guide and accelerate the search for errors. Our study of four PSEs covers mainstream industrial vendors (Google and NVIDIA) as well as academic products. We successfully uncover over 5K error-triggering inputs that generate incorrect simulation results spanning across the whole software stack of PSEs.
What problem does this paper attempt to address?
### Problems Addressed by the Paper
This paper aims to address the issue of errors in modern Physics Simulation Engines (PSE). Specifically, it proposes a fuzz testing framework named PHYFU, designed to detect logical errors in PSE during both forward and backward simulation stages.
#### Background and Motivation
1. **Applications of Physics Simulation Engines**:
- Modern PSEs support not only forward simulation but also backward simulation, providing gradient information to accelerate learning-based control tasks, such as robotic arm learning to grasp objects.
- PSEs are widely used in games, movies, virtual reality (VR), and robotics, holding high commercial value.
2. **Existing Problems**:
- Despite their widespread use in academia and industry, PSEs can produce erroneous simulation results, leading to negative consequences ranging from poor entertainment experiences to accidents in robotic manufacturing and surgical operations.
- PSEs typically contain hundreds of thousands of lines of code, covering a complex software stack that includes simulation algorithms, hardware acceleration modules, domain-specific languages (DSLs), and their compilers, making them highly complex and prone to errors.
#### Solution
1. **PHYFU Framework**:
- PHYFU is an automated, systematic fuzz testing framework specifically designed for modern PSEs, capable of detecting errors in a black-box environment.
- By generating and mutating initial states and testing based on physical laws (PLs), PHYFU can identify logical errors in both forward and backward simulation stages.
2. **Testing Methods**:
- **Forward Testing**: Based on the uniqueness of physical laws, ensuring that different initial states do not lead to the same final state.
- **Backward Testing**: Based on gradient optimization theory, ensuring that gradients correctly guide the optimization process, causing the objective function to converge to a minimum.
3. **Optimization and Feedback**:
- To improve testing efficiency, PHYFU employs a feedback-driven test input scheduling algorithm, prioritizing seeds that are more likely to trigger errors.
- The generated test inputs must satisfy actual physical constraints to ensure the validity of the tests.
#### Experiments and Results
- **Experimental Setup**: Targeting four mainstream PSEs (Brax, Warp, Taichi, and Nimble), covering various physical scenarios such as the simulation of spheres, robotic arms, and soft objects.
- **Experimental Results**: Over approximately 20 days of testing, PHYFU detected 5,932 inputs leading to erroneous simulation results, including over 20 inputs that caused crashes, revealing multiple hidden errors in the PSE software stack.
### Conclusion
By proposing the PHYFU framework, this paper systematically addresses the issue of error detection in modern physics simulation engines, enhancing the reliability and safety of PSEs. This work is significant for ensuring the correctness and stability of PSEs in various application scenarios.