Abstract:This paper proposes a novel deep reinforcement learning algorithm to perform automatic analysis and detection of gameplay issues in complex 3D navigation environments. The Curiosity-Conditioned Proximal Trajectories (CCPT) method combines curiosity and imitation learning to train agents to methodically explore in the proximity of known trajectories derived from expert demonstrations. We show how CCPT can explore complex environments, discover gameplay issues and design oversights in the process, and recognize and highlight them directly to game designers. We further demonstrate the effectiveness of the algorithm in a novel 3D navigation environment which reflects the complexity of modern AAA video games. Our results show a higher level of coverage and bug discovery than baselines methods, and it hence can provide a valuable tool for game designers to identify issues in game design automatically.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenges of automatic game testing and verification in modern 3D video games. Specifically, the authors propose a new deep reinforcement learning algorithm - Curiosity - Conditioned Proximal Trajectories (CCPT) for automatically analyzing and detecting game problems in complex 3D navigation environments. ### Problem Background 1. **Limitations of Manual Testing**: Modern video games are large - scale and complex, and it is difficult to comprehensively cover all possible game scenarios and problems by manual testing alone. 2. **Requirement for Automated Testing**: An automated method is needed to efficiently discover and identify bugs and design flaws in games, so as to reduce the burden of manual testing and improve testing efficiency. ### Paper Objectives The main objective of the paper is to develop intelligent agents that can automatically explore and analyze large - scale 3D environments. These agents can systematically discover and highlight problems and design omissions in games by combining imitation learning and curiosity - driven exploration. Specifically: - **Automatically Discover and Highlight Problems**: By training intelligent agents to explore near known trajectories, discover and mark bugs and design flaws in games. - **Improve Testing Coverage**: Compared with traditional methods, CCPT can explore in a wider area, thereby improving testing coverage and problem discovery rate. ### Method Overview The core of the CCPT method lies in its exploration - conditioned intrinsic reward function, which combines the curiosity module, the imitation learning module, and the navigation module: - **Curiosity Module**: Use the Random Network Distillation (RND) algorithm to encourage agents to explore states that have not been encountered or are less encountered. - **Imitation Learning Module**: Based on the Adversarial Motion Prior (AMP) algorithm, enable agents to imitate the trajectories demonstrated by experts. - **Navigation Module**: Utilize positional embeddings and 3D semantic occupancy maps to encode environmental information, helping agents better understand environmental states. ### Experimental Results The experimental results show that CCPT can effectively discover and highlight bugs and design flaws in games in complex 3D environments. Compared with baseline methods (such as using only imitation learning or curiosity - driven exploration), CCPT performs better in terms of testing coverage and problem discovery. ### Conclusion CCPT provides an effective automated game - testing tool that can help game developers and designers automatically identify and solve problems in games, significantly improving testing efficiency and quality. --- Through this method, CCPT can not only automatically discover bugs and design flaws in games, but also provide valuable feedback to game developers to help them improve game design.

CCPT: Automatic Gameplay Testing and Validation with Curiosity-Conditioned Proximal Trajectories

Self-play Reinforcement Learning with Comprehensive Critic in Computer Games

Large Scale Pursuit-Evasion under Collision Avoidance Using Deep Reinforcement Learning.

Inspector: Pixel-Based Automated Game Testing via Exploration, Detection, and Investigation

Efficient Exploration via First-Person Behavior Cloning Assisted Rapidly-Exploring Random Trees

Prototyping three key properties of specific curiosity in computational reinforcement learning

Taking the Scenic Route: Automatic Exploration for Videogames

SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World

Robust Imitation Learning for Automated Game Testing

Towards Informed Design and Validation Assistance in Computer Games Using Imitation Learning

Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning

Visual CPG-RL: Learning Central Pattern Generators for Visually-Guided Quadruped Locomotion

Deep Curiosity Search: Intra-Life Exploration Can Improve Performance on Challenging Deep Reinforcement Learning Problems

Navigates Like Me: Understanding How People Evaluate Human-Like AI in Video Games

Navigating the Human Maze: Real-Time Robot Pathfinding with Generative Imitation Learning

Preference-conditioned Pixel-based AI Agent For Game Testing

Deep Reinforcement Learning for Navigation in AAA Video Games

Learning to explore by reinforcement over high-level options

Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedback

Attention-based Curiosity-driven Exploration in Deep Reinforcement Learning

Predicting Game Difficulty and Engagement Using AI Players