Abstract:Dynamic Portfolio optimization is the process of distribution and rebalancing of a fund into different financial assets such as stocks, cryptocurrencies, etc, in consecutive trading periods to maximize accumulated profits or minimize risks over a time horizon. This field saw huge developments in recent years, because of the increased computational power and increased research in sequential decision making through control theory. Recently Reinforcement Learning(RL) has been an important tool in the development of sequential and dynamic portfolio optimization theory. In this paper, we design a Deep Reinforcement Learning(DRL) framework as an autonomous portfolio optimization agent consisting of a Latent Feature State Space(LFSS) Module for filtering and feature extraction of financial data which is used as a state space for deep RL model. We develop an extensive RL agent with high efficiency and performance advantages over several benchmarks and model-free RL agents used in prior work. The noisy and non-stationary behaviour of daily asset prices in the financial market is addressed through Kalman Filter. Autoencoders, ZoomSVD, and restricted Boltzmann machines were the models used and compared in the module to extract relevant time series features as state space. We simulate weekly data, with practical constraints and transaction costs, on a portfolio of S&P 500 stocks. We introduce a new benchmark based on technical indicator Kd-Index and Mean-Variance Model as compared to equal weighted portfolio used in most of the prior work. The study confirms that the proposed RL portfolio agent with state space function in the form of LFSS module gives robust results with an attractive performance profile over baseline RL agents and given benchmarks.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is Dynamic Portfolio Optimization. Specifically, the author aims to design a framework based on Deep Reinforcement Learning (DRL) and introduce the Latent Feature State Space (LFSS) module to filter and extract key features from financial data, thereby achieving more efficient portfolio optimization.
### Problem Background
Dynamic portfolio optimization refers to the allocation of funds to different financial assets (such as stocks, cryptocurrencies, etc.) during consecutive trading periods and continuously adjusting the proportions of these assets according to real - time financial information to maximize cumulative returns or minimize risks. In recent years, with the improvement of computing power and the research progress of sequential decision - making theory, especially the application of Reinforcement Learning (RL), this field has achieved remarkable development.
### Limitations of Existing Methods
Traditional methods such as Modern Portfolio Theory (MPT) have the limitations of static and linear calculations. Although methods such as dynamic programming and convex optimization can handle more complex situations, they rely on discrete action - space models and are not efficient in capturing market information. In addition, although some existing Deep Reinforcement Learning methods perform well in some aspects, they still face challenges when dealing with high - dimensional, noisy, and non - stationary financial market data.
### Core Contributions of the Paper
To solve the above problems, this paper proposes a Deep Reinforcement Learning framework that includes an LFSS module. Specifically:
1. **Kalman Filter**: Used to reduce the noise in asset price signals.
2. **Latent Feature Extraction Unit**: Including Autoencoders, ZoomSVD, and Restricted Boltzmann Machines (RBM), which are respectively used to extract low - dimensional time - series features.
3. **New Benchmark Indicators**: New benchmarks based on the technical indicator KD - Index and the mean - variance model are introduced to evaluate model performance.
### Main Objectives
By introducing the LFSS module, the author hopes to improve the existing Deep Reinforcement Learning framework in the following aspects:
- Extract more effective features and reduce the impact of noise.
- Handle high - dimensional data and avoid the curse of dimensionality.
- Provide better performance, surpassing existing benchmarks and other Deep Reinforcement Learning agents.
### Summary of Mathematical Formulas
- **State Transition Equation of Kalman Filter**:
\[
x_{t + 1}=F_{t}x_{t}+w_{t}, \quad w_{t}\sim N(0, Q)
\]
\[
y_{t}=G_{t}x_{t}+v_{t}, \quad v_{t}\sim N(0, R)
\]
- **Objective Function of Autoencoder**:
\[
\phi, \Omega=\arg\min_{\phi, \Omega}\|X - (\phi\circ\Omega)X\|^{2}
\]
\[
L = \|X - X'\|^{2}
\]
- **Matrix Decomposition of ZoomSVD**:
\[
A = U\Sigma V^{T}
\]
- **Energy Function of Restricted Boltzmann Machine**:
\[
E(v, h)=-h^{T}Wv - a^{T}v - b^{T}h
\]
Through these methods, the author aims to construct a more robust and efficient Deep Reinforcement Learning agent to deal with the complexity and uncertainty of the financial market.