Abstract:Actor-critic algorithms learn an explicit policy (actor), and an accompanying value function (critic). The actor performs actions in the environment, while the critic evaluates the actor's current policy. However, despite their stability and promising convergence properties, current actor-critic algorithms do not outperform critic-only ones in practice. We believe that the fact that the critic learns Q^pi, instead of the optimal Q-function Q*, prevents state-of-the-art robust and sample-efficient off-policy learning algorithms from being used. In this paper, we propose an elegant solution, the Actor-Advisor architecture, in which a Policy Gradient actor learns from unbiased Monte-Carlo returns, while being shaped (or advised) by the Softmax policy arising from an off-policy critic. The critic can be learned independently from the actor, using any state-of-the-art algorithm. Being advised by a high-quality critic, the actor quickly and robustly learns the task, while its use of the Monte-Carlo return helps overcome any bias the critic may have. In addition to a new Actor-Critic formulation, the Actor-Advisor, a method that allows an external advisory policy to shape a Policy Gradient actor, can be applied to many other domains. By varying the source of advice, we demonstrate the wide applicability of the Actor-Advisor to three other important subfields of RL: safe RL with backup policies, efficient leverage of domain knowledge, and transfer learning in RL. Our experimental results demonstrate the benefits of the Actor-Advisor compared to state-of-the-art actor-critic methods, illustrate its applicability to the three other application scenarios listed above, and show that many important challenges of RL can now be solved using a single elegant solution.

A Supervised Goal Directed Algorithm in Economical Choice Behaviour: An Actor-Critic Approach

Model Reference Output Feedback Control Using Episodic Natural Actor-Critic

Decision-aid or Controller? Steering Human Decision Makers with Algorithms

Policy adjustment in a dynamic economic game

On the Theory of Risk-Aware Agents: Bridging Actor-Critic and Economics

Learning from Zero: How to Make Consumption-Saving Decisions in a Stochastic Environment with an AI Algorithm

The hippocampal-striatal circuit for goal-directed and habitual choice

The Actor-Advisor: Policy Gradient With Off-Policy Advice

Reinforcement Learning in Economics and Finance

Balancing Rational and Other-Regarding Preferences in Cooperative-Competitive Environments

Reinforcement Learning for Continuous-Time Optimal Execution: Actor-Critic Algorithm and Error Analysis

Exploring Pessimism and Optimism Dynamics in Deep Reinforcement Learning

PAC-Bayesian Soft Actor-Critic Learning

Opponent actor learning (OpAL): Modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive.

Apparently Irrational Choice as Optimal Sequential Decision Making

Neurocomputations of strategic behavior: From iterated to novel interactions

Brain signals of a Surprise-Actor-Critic model: Evidence for multiple learning modules in human decision making

Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPs

Explorer-Actor-Critic: Better Actors for Deep Reinforcement Learning

Topological Guided Actor-Critic Modular Learning of Continuous Systems with Temporal Objectives

An Actor-Critic Method for Simulation-Based Optimization