Abstract:High-frequency trading (HFT) uses computer algorithms to make trading decisions in short time scales (e.g., second-level), which is widely used in the Cryptocurrency (Crypto) market (e.g., Bitcoin). Reinforcement learning (RL) in financial research has shown stellar performance on many quantitative trading tasks. However, most methods focus on low-frequency trading, e.g., day-level, which cannot be directly applied to HFT because of two challenges. First, RL for HFT involves dealing with extremely long trajectories (e.g., 2.4 million steps per month), which is hard to optimize and evaluate. Second, the dramatic price fluctuations and market trend changes of Crypto make existing algorithms fail to maintain satisfactory performance. To tackle these challenges, we propose an Efficient hieArchical Reinforcement learNing method for High Frequency Trading (EarnHFT), a novel three-stage hierarchical RL framework for HFT. In stage I, we compute a Q-teacher, i.e., the optimal action value based on dynamic programming, for enhancing the performance and training efficiency of second-level RL agents. In stage II, we construct a pool of diverse RL agents for different market trends, distinguished by return rates, where hundreds of RL agents are trained with different preferences of return rates and only a tiny fraction of them will be selected into the pool based on their profitability. In stage III, we train a minute-level router which dynamically picks a second-level agent from the pool to achieve stable performance across different markets. Through extensive experiments in various market trends on Crypto markets in a high-fidelity simulation trading environment, we demonstrate that EarnHFT significantly outperforms 6 state-of-art baselines in 6 popular financial criteria, exceeding the runner-up by 30% in profitability.

A Framework for Empowering Reinforcement Learning Agents with Causal Analysis: Enhancing Automated Cryptocurrency Trading

Extending Deep Reinforcement Learning Frameworks in Cryptocurrency Market Making

A Scalable Reinforcement Learning-based System Using On-Chain Data for Cryptocurrency Portfolio Management

Reinforcement Learning Framework for Quantitative Trading

Optimizing Portfolio with Two-Sided Transactions and Lending: A Reinforcement Learning Framework

Multi-Agent Deep Reinforcement Learning With Progressive Negative Reward for Cryptocurrency Trading

Reinforcement Learning Pair Trading: A Dynamic Scaling approach

Deep Robust Reinforcement Learning for Practical Algorithmic Trading

A Deep Reinforcement Learning Approach for Trading Optimization in the Forex Market with Multi-Agent Asynchronous Distribution

UNSURE - A machine learning approach to cryptocurrency trading

Deep Reinforcement Learning For Trading - A Critical Survey

Optimizing Automated Trading Systems with Deep Reinforcement Learning

FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance

EarnHFT: Efficient Hierarchical Reinforcement Learning for High Frequency Trading

Automatic Financial Trading Agent for Low-risk Portfolio Management using Deep Reinforcement Learning

A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem

Modelling crypto markets by multi-agent reinforcement learning

Deep reinforcement learning on a multi-asset environment for trading

Cooperative Multi-Agent Reinforcement Learning Framework for Scalping Trading

Deep Reinforcement Learning in Quantitative Algorithmic Trading: A Review

An Ensemble Method of Deep Reinforcement Learning for Automated Cryptocurrency Trading