Abstract:High-frequency trading (HFT) that executes algorithmic trading in short time scales, has recently occupied the majority of cryptocurrency market. Besides traditional quantitative trading methods, reinforcement learning (RL) has become another appealing approach for HFT due to its terrific ability of handling high-dimensional financial data and solving sophisticated sequential decision-making problems, \emph{e.g.,} hierarchical reinforcement learning (HRL) has shown its promising performance on second-level HFT by training a router to select only one sub-agent from the agent pool to execute the current transaction. However, existing RL methods for HFT still have some defects: 1) standard RL-based trading agents suffer from the overfitting issue, preventing them from making effective policy adjustments based on financial context; 2) due to the rapid changes in market conditions, investment decisions made by an individual agent are usually one-sided and highly biased, which might lead to significant loss in extreme markets. To tackle these problems, we propose a novel Memory Augmented Context-aware Reinforcement learning method On HFT, \emph{a.k.a.} MacroHFT, which consists of two training phases: 1) we first train multiple types of sub-agents with the market data decomposed according to various financial indicators, specifically market trend and volatility, where each agent owns a conditional adapter to adjust its trading policy according to market conditions; 2) then we train a hyper-agent to mix the decisions from these sub-agents and output a consistently profitable meta-policy to handle rapid market fluctuations, equipped with a memory mechanism to enhance the capability of decision-making. Extensive experiments on various cryptocurrency markets demonstrate that MacroHFT can achieve state-of-the-art performance on minute-level trading tasks.

What problem does this paper attempt to address?

The paper attempts to address several key challenges in high-frequency trading (HFT): 1. **Overfitting Issue**: Existing reinforcement learning (RL)-based trading agents tend to overfit when dealing with financial contexts, limiting their ability to effectively adjust strategies based on market conditions. 2. **Decision Bias Due to Rapid Market Changes**: Due to the rapid changes in market conditions, investment decisions made by individual agents are often one-sided and highly biased, which can lead to significant losses under extreme market conditions. 3. **Neglect of Market Volatility**: Most existing methods treat the cryptocurrency market as a unified and static entity or distinguish market conditions only based on market trends, ignoring market volatility. This neglect is particularly pronounced in the highly dynamic cryptocurrency market, leading to poor risk management and reduced professionalism and effectiveness of trading strategies. To address these challenges, the authors propose a new memory-augmented context-aware reinforcement learning method—MacroHFT (Memory Augmented Context-aware Reinforcement Learning On High Frequency Trading), focusing on minute-level cryptocurrency trading and incorporating macro market information to assist trading decisions. Specifically, the workflow of MacroHFT mainly includes two stages: 1. **First Stage**: MacroHFT decomposes the cryptocurrency market into different categories based on trend and volatility indicators, then trains multiple diversified sub-agents under different market dynamics. Each sub-agent has a conditional adapter that can adjust its trading strategy according to market conditions. 2. **Second Stage**: MacroHFT trains a super-agent as a mixture of policies of all sub-agents, leveraging their profitability under different market dynamics. The super-agent is equipped with a memory mechanism that can learn from recent experiences, generating stable trading strategies while maintaining a quick response to extreme volatility. Through these two stages of training, MacroHFT aims to overcome the shortcomings of existing methods and improve the performance of high-frequency trading in the cryptocurrency market. Experimental results show that MacroHFT significantly outperforms many existing state-of-the-art baseline methods in various popular cryptocurrency markets.

MacroHFT: Memory Augmented Context-aware Reinforcement Learning On High Frequency Trading

EarnHFT: Efficient Hierarchical Reinforcement Learning for High Frequency Trading

QF-TraderNet: Intraday Trading via Deep Reinforcement With Quantum Price Levels Based Profit-And-Loss Control

Multi-Agent Deep Reinforcement Learning for High-Frequency Multi-Market Making

Quantum Finance and Fuzzy Reinforcement Learning-Based Multi-agent Trading System

DeepScalper: A Risk-Aware Reinforcement Learning Framework to Capture Fleeting Intraday Trading Opportunities

Harnessing Deep Q-Learning for Enhanced Statistical Arbitrage in High-Frequency Trading: A Comprehensive Exploration

Algorithmic Currency Trading Based On Reinforcement Learning Combining Action Shaping And Advantage Function Shaping

Deep Reinforcement Learning for Quantitative Trading

Extending Deep Reinforcement Learning Frameworks in Cryptocurrency Market Making

Financial Trading as a Game: A Deep Reinforcement Learning Approach

Deep Reinforcement Learning for High-Frequency Market Making

High-Frequency Quantitative Trading of Digital Currencies Based on Fusion of Deep Reinforcement Learning Models with Evolutionary Strategies

MCTG: Multi-frequency Continuous-Share Trading Algorithm with GARCH Based on Deep Reinforcement Learning

Multi-agent reinforcement learning in a realistic limit order book market simulation

Deep Stock Trading: A Hierarchical Reinforcement Learning Framework for Portfolio Optimization and Order Execution

Auto Uning of Price Prediction Models for High-Frequency Trading Via Reinforcement Learning

Deep reinforcement learning on a multi-asset environment for trading

Deep Reinforcement Learning for Active High Frequency Trading

FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design

MacMic: Executing Iceberg Orders Via Hierarchical Reinforcement Learning