Learning Diverse Policies with Soft Self-Generated Guidance

Guojian Wang, Faguo Wu, Xiao Zhang, Jianxiang Liu

DOI: https://doi.org/10.1155/2023/4705291

2024-02-07

Abstract:Reinforcement learning (RL) with sparse and deceptive rewards is challenging because non-zero rewards are rarely obtained. Hence, the gradient calculated by the agent can be stochastic and without valid information. Recent studies that utilize memory buffers of previous experiences can lead to a more efficient learning process. However, existing methods often require these experiences to be successful and may overly exploit them, which can cause the agent to adopt suboptimal behaviors. This paper develops an approach that uses diverse past trajectories for faster and more efficient online RL, even if these trajectories are suboptimal or not highly rewarded. The proposed algorithm combines a policy improvement step with an additional exploration step using offline demonstration data. The main contribution of this paper is that by regarding diverse past trajectories as guidance, instead of imitating them, our method directs its policy to follow and expand past trajectories while still being able to learn without rewards and approach optimality. Furthermore, a novel diversity measurement is introduced to maintain the team's diversity and regulate exploration. The proposed algorithm is evaluated on discrete and continuous control tasks with sparse and deceptive rewards. Compared with the existing RL methods, the experimental results indicate that our proposed algorithm is significantly better than the baseline methods regarding diverse exploration and avoiding local optima.

Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The paper mainly discusses the challenges faced by reinforcement learning (RL) in handling sparse and deceptive rewards. In such cases, non-zero rewards are rarely obtained, resulting in gradients computed by the agent that may have randomness and invalid information. Although there are studies that use a past memory buffer to improve learning efficiency, these methods often require successful experiences and over-rely on them, which may lead to the adoption of suboptimal behaviors by the agent.

The paper proposes a new approach that utilizes diverse past trajectories for faster and more efficient online RL learning, even when the past trajectories are suboptimal or have low rewards. This approach combines policy improvement steps and additional exploration steps using offline demonstration data. The key contribution is to treat the diverse past trajectories as guidance rather than imitation, guiding the policy to follow and extend the past trajectories while still being able to learn and approach the optimal solution in the absence of rewards. Additionally, a new diversity measure is introduced to maintain diversity and regulate exploration.

Experiments show that compared to existing RL methods, this algorithm performs significantly better in terms of diverse exploration and avoiding local optima in discrete and continuous control tasks with sparse and deceptive rewards. This method addresses the performance degradation issue caused by over-reliance on successful experiences in sparse reward environments, and improves learning efficiency by indirectly driving exploration through imperfect trajectories.