Abstract:With increasing numbers of vulnerabilities exposed on the internet, autonomous penetration testing (pentesting) has emerged as an emerging research area, while reinforcement learning (RL) is a natural fit for studying autonomous pentesting. Previous research in RL-based autonomous pentesting mainly focused on enhancing agents' learning efficacy within abstract simulated training environments. They overlooked the applicability and generalization requirements of deploying agents' policies in real-world environments that differ substantially from their training settings. In contrast, for the first time, we shift focus to the pentesting agents' ability to generalize across unseen real environments. For this purpose, we propose a Generalizable Autonomous Pentesting framework (namely GAP) for training agents capable of drawing inferences from one to another -- a key requirement for the broad application of autonomous pentesting and a hallmark of human intelligence. GAP introduces a Real-to-Sim-to-Real pipeline with two key methods: domain randomization and meta-RL learning. Specifically, we are among the first to apply domain randomization in autonomous pentesting and propose a large language model-powered domain randomization method for synthetic environment generation. We further apply meta-RL to improve the agents' generalization ability in unseen environments by leveraging the synthetic environments. The combination of these two methods can effectively bridge the generalization gap and improve policy adaptation performance. Experiments are conducted on various vulnerable virtual machines, with results showing that GAP can (a) enable policy learning in unknown real environments, (b) achieve zero-shot policy transfer in similar environments, and (c) realize rapid policy adaptation in dissimilar environments.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the problem of insufficient generalization ability in autonomous penetration testing (pentesting). Specifically: 1. **Limitations of traditional methods**: - Previous research has mainly focused on improving the learning efficiency of agents in abstract simulated training environments, while ignoring the applicability and generalization requirements of these agents when deployed in real - world environments. - Automated penetration testing is a dynamic sequential decision - making process, and Reinforcement Learning (RL) is an ideal method for optimizing such decisions. However, RL algorithms usually require a large number of training samples, which is impractical in actual penetration testing because interacting with the real - world environment is both time - consuming and risky. 2. **Reality Gap and Generalization Gap**: - The Reality Gap refers to the difference between the simulated environment and the real - world environment. Even if the vulnerabilities are the same, different host configurations will lead to changes in the observation results, thus affecting the transfer learning performance of the agents. - The Generalization Gap means that the agents over - fit in the training environment, resulting in poor performance in unseen real - world scenarios. This is because the diversity of the training environment is limited, while the real - world environment is unpredictable and diverse. 3. **Research objectives**: - For the first time, this paper focuses on the generalization ability of agents in unseen real - world environments and proposes a Generalizable Autonomous Pentesting framework (GAP) to achieve the ability to infer from one instance to another, similar to the characteristics of human intelligence. - Specific objectives include: - Achieving zero - shot policy transfer in similar environments. - Achieving few - shot policy adaptation in different environments, thereby improving the overall learning efficiency. 4. **Solutions**: - **Domain Randomization**: By highly randomizing the rendering settings of the simulated training set, increase the diversity and complexity of the environment and prevent agents from over - fitting specific observed features. - **Meta - Reinforcement Learning (Meta - RL)**: Utilize the generated simulated training environment to extract inductive biases and enhance the generalization ability of agents in unseen environments. ### Summary By proposing the GAP framework and combining domain randomization and meta - reinforcement learning, this paper aims to solve the generalization ability and applicability problems of existing autonomous penetration testing methods in real - world applications, enabling agents to perform penetration testing more effectively in unknown environments.

Towards Generalizable Autonomous Penetration Testing via Domain Randomization and Meta-Reinforcement Learning

MARNet: Backdoor Attacks Against Cooperative Multi-Agent Reinforcement Learning

Behaviour-diverse automatic penetration testing: a coverage-based deep reinforcement learning approach

SetTron: Towards Better Generalisation in Penetration Testing with Reinforcement Learning

DynPen: Automated Penetration Testing in Dynamic Network Scenarios Using Deep Reinforcement Learning

EPPTA: Efficient Partially Observable Reinforcement Learning Agent for Penetration Testing Applications

GAIL-PT: An Intelligent Penetration Testing Framework with Generative Adversarial Imitation Learning

Knowledge-Informed Auto-Penetration Testing Based on Reinforcement Learning with Reward Machine

An Automated Penetration Testing Framework Based on Hierarchical Reinforcement Learning

Transferable Adversarial Attacks on Deep Reinforcement Learning with Domain Randomization

PentestAgent: Incorporating LLM Agents to Automated Penetration Testing

A Hierarchical Deep Reinforcement Learning Model with Expert Prior Knowledge for Intelligent Penetration Testing

Hierarchical reinforcement learning for efficient and effective automated penetration testing of large networks

Raijū: Reinforcement Learning-Guided Post-Exploitation for Automating Security Assessment of Network Systems

Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentation

Evaluation of Reinforcement Learning for Autonomous Penetration Testing using A3C, Q-learning and DQN

Autonomous Penetration Testing Based on Improved Deep Q-Network

A Layered Reference Model for Penetration Testing with Reinforcement Learning and Attack Graphs

PentestGPT: An LLM-empowered Automatic Penetration Testing Tool

Towards Automated Penetration Testing: Introducing LLM Benchmark, Analysis, and Improvements