Abstract:Financial portfolio management investment policies computed quantitatively by modern portfolio theory techniques like the Markowitz model rely on a set on assumptions that are not supported by data in high volatility markets. Hence, quantitative researchers are looking for alternative models to tackle this problem. Concretely, portfolio management is a problem that has been successfully addressed recently by Deep Reinforcement Learning (DRL) approaches. In particular, DRL algorithms train an agent by estimating the distribution of the expected reward of every action performed by an agent given any financial state in a simulator. However, these methods rely on Deep Neural Networks model to represent such a distribution, that although they are universal approximator models, they cannot explain its behaviour, given by a set of parameters that are not interpretable. Critically, financial investors policies require predictions to be interpretable, so DRL agents are not suited to follow a particular policy or explain their actions. In this work, we developed a novel Explainable Deep Reinforcement Learning (XDRL) approach for portfolio management, integrating the Proximal Policy Optimization (PPO) with the model agnostic explainable techniques of feature importance, SHAP and LIME to enhance transparency in prediction time. By executing our methodology, we can interpret in prediction time the actions of the agent to assess whether they follow the requisites of an investment policy or to assess the risk of following the agent suggestions. To the best of our knowledge, our proposed approach is the first explainable post hoc portfolio management financial policy of a DRL agent. We empirically illustrate our methodology by successfully identifying key features influencing investment decisions, which demonstrate the ability to explain the agent actions in prediction time.

End-to-End Policy Gradient Method for POMDPs and Explainable Agents

Learning Interpretable Policies in Hindsight-Observable POMDPs through Partially Supervised Reinforcement Learning

ODE-based Recurrent Model-free Reinforcement Learning for POMDPs

Leveraging Counterfactual Paths for Contrastive Explanations of POMDP Policies

Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods

Recurrent Natural Policy Gradient for POMDPs

Learning Explainable and Better Performing Representations of POMDP Strategies

Explanation through Reward Model Reconciliation using POMDP Tree Search

Goal-oriented inference of environment from redundant observations

Explainable Post hoc Portfolio Management Financial Policy of a Deep Reinforcement Learning agent

Provable Representation with Efficient Planning for Partial Observable Reinforcement Learning

Learning for Decentralized Control of Multiagent Systems in Large, Partially-Observable Stochastic Environments

Latent-Conditioned Policy Gradient for Multi-Objective Deep Reinforcement Learning

Explainable Finite-Memory Policies for Partially Observable Markov Decision Processes

Policy Optimization with Smooth Guidance Learned from State-Only Demonstrations

Policy Optimization with Model-based Explorations

Flow-based Recurrent Belief State Learning for POMDPs

A Scalable Model-Free Recurrent Neural Network Framework for Solving POMDPs

Embed to Control Partially Observed Systems: Representation Learning with Provable Sample Efficiency

PP-PG: Combining Parameter Perturbation with Policy Gradient Methods for Effective and Efficient Explorations in Deep Reinforcement Learning

On Improving Deep Reinforcement Learning for POMDPs