Abstract:While research of reinforcement learning applied to financial markets predominantly concentrates on finding optimal behaviours, it is worth to realize that the reinforcement learning returns $G_t$ and state value functions themselves are of interest and play a pivotal role in the evaluation of assets. Instead of focussing on the more complex task of finding optimal decision rules, this paper studies and applies the power of distributional state value functions in the context of financial market valuation and machine learning based trading algorithms. Accurate and trustworthy estimates of the distributions of $G_t$ provide a competitive edge leading to better informed decisions and more optimal behaviour. Herein, ideas from predictive knowledge and deep reinforcement learning are combined to introduce a novel family of models called CDG-Model, resulting in a highly flexible framework and intuitive approach with minimal assumptions regarding underlying distributions. The models allow seamless integration of typical financial modelling pitfalls like transaction costs, slippage and other possible costs or benefits into the model calculation. They can be applied to any kind of trading strategy or asset class. The frameworks introduced provide concrete business value through their potential in market valuation of single assets and portfolios, in the comparison of strategies as well as in the improvement of market timing. They can positively impact the performance and enhance the learning process of existing or new trading algorithms. They are of interest from a scientific point-of-view and open up multiple areas of future research. Initial implementations and tests were performed on real market data. While the results are promising, applying a robust statistical framework to evaluate the models in general remains a challenge and further investigations are needed.

Optional Feature Vector Generation for Linear Value Function Approximation with Binary Features

Greedy Feature Replacement for Online Value Function Approximation

Provably Efficient Reinforcement Learning with General Value Function Approximation.

Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learning

Randomized Exploration for Reinforcement Learning with General Value Function Approximation

Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learning

Foresee then Evaluate: Decomposing Value Estimation with Latent Future Prediction.

Visible-hidden Hybrid Automatic Feature Engineering Via Multi-Agent Reinforcement Learning

Provably Efficient Reinforcement Learning with Linear Function Approximation

The Optimal Approximation Factors in Misspecified Off-Policy Value Function Estimation

Value-Gradient Iteration with Quadratic Approximate Value Functions

Exploiting Distributional Value Functions for Financial Market Valuation, Enhanced Feature Creation and Improvement of Trading Algorithms

Interpretable Option Discovery using Deep Q-Learning and Variational Autoencoders

Random Fourier Feature Kernel Recursive Least Squares.

Role-oriented Binary Grey Wolf Optimizer Using Foraging-Following and Lévy Flight for Feature Selection

Hierarchical Universal Value Function Approximators

Adaptive Exploration for Data-Efficient General Value Function Evaluations

Tractable and Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation

A Double-Variational Bayesian Framework in Random Fourier Features for Indefinite Kernels

An Enhanced Fast Encoding Method for Vector Quantization by Constructing New Feature Based on the Variances of Subvectors