Abstract:In order to cope with ever-evolving and increasing cyber threats, intrusion detection systems have become a crucial component of cyber security. Compared with signature-based intrusion detection methods, anomaly-based methods typically employ machine learning techniques to train detection models and possess the capability to discover unknown attacks. However, intrusion detection methods face the challenge of low detection rates for minority class attacks due to imbalanced data distributions. Traditional intrusion detection algorithms address this issue by resampling or generating synthetic data. Additionally, reinforcement learning, as a machine learning method that interacts with the environment to obtain feedback and improve performance, is gradually being considered for application in the field of intrusion detection. This paper proposes a reinforcement-learning-based intrusion detection method that innovatively uses adaptive sample distribution dual-experience replay to enhance a reinforcement learning algorithm, aiming to effectively address the issue of imbalanced sample distribution. We have also developed a reinforcement learning environment specifically designed for intrusion detection tasks. Experimental results demonstrate that the proposed model achieves favorable performance on the NSL-KDD, AWID, and CICIoT2023 datasets, effectively dealing with imbalanced data and showing better classification performance in detecting minority attacks.
What problem does this paper attempt to address?
### Problems the paper attempts to solve
This paper aims to solve the problem of intrusion detection in network security, especially in the case of unbalanced data sample distribution. Specifically, traditional intrusion detection methods have a low detection rate when dealing with minority - class attacks because the number of normal samples is far greater than that of abnormal samples, making it difficult for the model to learn effective classification patterns from abnormal samples. To solve this problem, the paper proposes an intrusion detection method based on adaptive sample - distribution double - experience - replay reinforcement learning.
### Main contributions
1. **Propose a new intrusion detection method**:
- Based on an improved reinforcement learning algorithm, a double - experience - replay buffer is designed as the data source for experience replay, and the weights of each classification category are adaptively adjusted to adjust the proportion of samples of each category in the replay buffer, thus effectively dealing with unbalanced datasets and achieving good detection performance.
2. **Develop a reinforcement learning environment suitable for intrusion detection tasks**:
- This environment provides a general - purpose interface for training and evaluating the performance of agents, providing a basis for applying reinforcement learning to intrusion detection tasks in different scenarios.
3. **Experimental verification**:
- Experiments were carried out on three datasets, namely NSL - KDD, AWID and CICIoT2023. The results show that the proposed model can efficiently detect and identify network intrusion behaviors. Especially when dealing with unbalanced data, this method shows better classification performance in detecting minority - class attacks.
### Method overview
The method proposed in the paper is based on adaptive sample - distribution double - experience - replay reinforcement learning and mainly includes the following parts:
1. **Data preparation**:
- Convert the original data into a format that meets the input requirements of the model, and perform data cleaning, feature selection, and data transformation and encoding to improve the training and prediction performance of the model.
2. **Reinforcement learning environment**:
- Provide the necessary conditions, including action space, state space, reward feedback, state - transition model and task - termination conditions, etc., so that the agent can interact with the outside world in the environment, obtain necessary information for perception and action.
3. **Detection model**:
- The detection model, as an agent in reinforcement learning, receives the environmental state (i.e., network traffic information), extracts network features, identifies potential intrusion behaviors, and classifies them. Through iterative experiments and exploration of the environment, the model continuously optimizes itself to obtain the optimal strategy.
### Specific design of the reinforcement learning environment
- **State space**: Consider network traffic features such as packet size, network protocol and transmission information as states to form a mixed state space.
- **Action space**: Be responsible for marking the incoming network traffic data as intrusion, that is, classifying it as normal or attack.
- **Reward function**: Give positive or negative rewards according to whether the classification action of the agent is consistent with the actual result.
- **State transition**: Describe the pattern of environmental change after the agent takes action.
- **Termination condition**: Define the conditions for the end of the task, for example, stop interacting with the environment after processing a predetermined number of traffic samples.
### Adaptive sample - distribution double - experience - replay reinforcement learning algorithm
This algorithm is improved on the basis of DQN. A second experience - replay buffer is introduced, and the experience distribution in it is based on the weights of samples of each category to more accurately reflect the real distribution of data samples. Through this method, the model can maintain high detection performance when dealing with unbalanced datasets.
In conclusion, the method proposed in the paper is not only innovative in theory but also shows good performance in practical applications, providing new ideas and solutions for research in the field of intrusion detection.