Abstract:Policy diversity, encompassing the variety of policies an agent can adopt, enhances reinforcement learning (RL) success by fostering more robust, adaptable, and innovative problem-solving in the environment. The environment in which standard RL operates is usually modeled with a Markov Decision Process (MDP) as the theoretical foundation. However, in many real-world scenarios, the rewards depend on an agent's history of states and actions leading to a non-MDP. Under the premise of policy diffusion initialization, non-MDPs may have unstructured expanding solution space due to varying historical information and temporal dependencies. This results in solutions having non-equivalent closed forms in non-MDPs. In this paper, deriving diverse solutions for non-MDPs requires policies to break through the boundaries of the current solution space through gradual dispersion. The goal is to expand the solution space, thereby obtaining more diverse policies. Specifically, we first model the sequences of states and actions by a transformer-based method to learn policy embeddings for dispersion in the solution space, since the transformer has advantages in handling sequential data and capturing long-range dependencies for non-MDP. Then, we stack the policy embeddings to construct a dispersion matrix as the policy diversity measure to induce the policy dispersion in the solution space and obtain a set of diverse policies. Finally, we prove that if the dispersion matrix is positive definite, the dispersed embeddings can effectively enlarge the disagreements across policies, yielding a diverse expression for the original policy embedding distribution. Experimental results of both non-MDP and MDP environments show that this dispersion scheme can obtain more expressive diverse policies via expanding the solution space, showing more robust performance than the recent learning baselines.

Diverse Exploration via Conjugate Policies for Policy Gradient Methods

Diverse Exploration for Fast and Safe Policy Improvement

Behind the Myth of Exploration in Policy Gradients

Non-local Policy Optimization via Diversity-regularized Collaborative Exploration

DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization

Careful at Estimation and Bold at Exploration

Efficient sample reuse in policy gradients with parameter-based exploration

Learning Diverse Policies with Soft Self-Generated Guidance

PP-PG: Combining Parameter Perturbation with Policy Gradient Methods for Effective and Efficient Explorations in Deep Reinforcement Learning

Boltzmann Exploration for Deterministic Policy Optimization.

Curious Explorer: a Provable Exploration Strategy in Policy Learning

Exploration in policy optimization through multiple paths

Improving exploration in policy gradient search: Application to symbolic optimization

Adaptive-Gradient Policy Optimization: Enhancing Policy Learning in Non-Smooth Differentiable Simulations

Diversifying Policies With Non-Markov Dispersion to Expand the Solution Space

Stabilizing Policy Gradients for Stochastic Differential Equations via Consistency with Perturbation Process

Exploring Policy Diversity in Parallel Actor-Critic Learning.

A Scalable Derivative-free Exploration Approach for Reinforcement Learning

Reinforcement Learning with Derivative-Free Exploration

Policy Gradient from Demonstration and Curiosity

Policy Manifold Search for Improving Diversity-based Neuroevolution