Abstract:Dynamic Difficulty Adjustment (DDA) is a viable approach to enhance a player's experience in video games. Recently, Reinforcement Learning (RL) methods have been employed for DDA in non-competitive games; nevertheless, they rely solely on discrete state-action space with a small search space. In this paper, we propose a continuous RL-based DDA methodology for a visual working memory (VWM) game to handle the complex search space for the difficulty of memorization. The proposed RL-based DDA tailors game difficulty based on the player's score and game difficulty in the last trial. We defined a continuous metric for the difficulty of memorization. Then, we consider the task difficulty and the vector of difficulty-score as the RL's action and state, respectively. We evaluated the proposed method through a within-subject experiment involving 52 subjects. The proposed approach was compared with two rule-based difficulty adjustment methods in terms of player's score and game experience measured by a questionnaire. The proposed RL-based approach resulted in a significantly better game experience in terms of competence, tension, and negative and positive affect. Players also achieved higher scores and win rates. Furthermore, the proposed RL-based DDA led to a significantly less decline in the score in a 20-trial session.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to achieve Dynamic Difficulty Adjustment (DDA) in the Visual Working Memory (VWM) game to enhance players' gaming experience. Specifically, the author proposes a DDA method based on Continuous Reinforcement Learning (RL), aiming to handle the complex search space of memory difficulty and customize the game difficulty according to the player's score in the previous trial and the game difficulty. ### Main problems 1. **Limitations of existing methods**: Most of the current DDA methods based on RL rely on discrete state - action spaces, which limits their ability to handle complex search spaces. 2. **Measurement of memory difficulty**: Existing research usually only considers the number of items to be remembered as a difficulty indicator, while ignoring other factors that may affect difficulty, such as the visual load and distribution of items. ### Solutions 1. **Continuous measurement of memory difficulty**: The author defines a continuous memory difficulty measurement to quantify the task difficulty by linearly combining three features (the number of targets \(n_t\), the number of connected components \(n_c\), and the distribution \(d\)): \[ \text{Difficulty} = \alpha_1 n_t+\alpha_2 n_c + \alpha_3 d \] where \(\alpha_1, \alpha_2, \alpha_3\) are weighting coefficients used to balance the influence of each feature on the difficulty. 2. **DDA based on continuous RL**: The author uses the Proximal Policy Optimization (PPO) algorithm to train the RL system. The system is preliminarily trained by simulating human players and then fine - tuned by real players. The state of the RL system consists of the task difficulty vector and the player's score, and the action is to determine the difficulty of the next memory task. ### Experiments and results 1. **Experimental design**: The author conducted an experiment with 52 participants, comparing the proposed RL - based DDA method with two rule - based DDA methods in terms of gaming experience and scores. 2. **Experimental results**: - **Gaming experience**: The RL - based DDA method is significantly superior to the rule - based methods in terms of competitiveness, tension, positive and negative emotions. - **Scores and win rates**: Participants obtained higher average scores and win rates under the RL - based DDA method. - **Score decline**: In 20 trials, the score decline caused by the RL - based DDA method is significantly smaller than that of the rule - based methods. ### Conclusions The method proposed by the author can not only effectively handle the complex search space of memory difficulty but also dynamically adjust the game difficulty according to the player's performance, thereby enhancing the player's gaming experience and performance. In addition, the proposed continuous memory difficulty measurement also performs well in reducing score decline, although its advantage in gaming experience is less significant. ### Future work 1. **Further optimize the model**: Consider including the pictures of memory tasks as part of the action and incorporating different dimensions of the player model into the state to provide a more personalized gaming experience. 2. **Enhance the difficulty measurement**: Use game data to adjust the covariates in the difficulty measurement to further optimize the model. Through these improvements, the author hopes to verify and apply the DDA method based on continuous RL in a wider range of scenarios.

Continuous Reinforcement Learning-based Dynamic Difficulty Adjustment in a Visual Working Memory Game

Dynamic difficulty adjustment using deep reinforcement learning: A review

DL-DDA -- Deep Learning based Dynamic Difficulty Adjustment with UX and Gameplay constraints

Dynamic difficulty adjustment approaches in video games: a systematic literature review

Exploring Dynamic Difficulty Adjustment in Videogames

Personalized Dynamic Difficulty Adjustment -- Imitation Learning Meets Reinforcement Learning

Dynamic Difficulty Adaptation Based on Stress Detection for a Virtual Reality Video Game: A Pilot Study

Rethinking dynamic difficulty adjustment for video game design

Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning

Colwell's Castle Defence: A Custom Game Using Dynamic Difficulty Adjustment to Increase Player Enjoyment

Player Dominance Adjustment in Games

Fast Game Content Adaptation Through Bayesian-based Player Modelling

Adversarial Autoencoder and Multi-Armed Bandit for Dynamic Difficulty Adjustment in Immersive Virtual Reality for Rehabilitation: Application to Hand Movement

A genetic algorithm-based method to modulate the difficulty of serious games along consecutive robot-assisted therapy sessions

Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsal

DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization

An Extensive-Form Game Paradigm for Visual Field Testing Via Deep Reinforcement Learning.

Accelerating Reinforcement Learning using EEG-based implicit human feedback

A data-driven approach for online adaptation of game difficulty

Reinforcement Learning for Dynamic Memory Allocation

Investigating Perceived and Mechanical Challenge in Games Through Cognitive Activity