Cybersecurity Software Tool Evaluation Using a 'Perfect' Network Model

Jeremy Straub

2024-09-14

Abstract:Cybersecurity software tool evaluation is difficult due to the inherently adversarial nature of the field. A penetration testing (or offensive) tool must be tested against a viable defensive adversary and a defensive tool must, similarly, be tested against a viable offensive adversary. Characterizing the tool's performance inherently depends on the quality of the adversary, which can vary from test to test. This paper proposes the use of a 'perfect' network, representing computing systems, a network and the attack pathways through it as a methodology to use for testing cybersecurity decision-making tools. This facilitates testing by providing a known and consistent standard for comparison. It also allows testing to include researcher-selected levels of error, noise and uncertainty to evaluate cybersecurity tools under these experimental conditions.

Cryptography and Security

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the difficulty in evaluating network security software tools. Specifically, due to the adversarial nature of the network security field, it is very challenging to evaluate the effectiveness of penetration - testing (offensive) tools and defensive tools. A penetration - testing tool must be tested in the face of effective defense measures, while a defensive tool needs to be tested when facing effective attack means. The performance evaluation of tools depends on the quality of opponents, and this quality may vary in different tests.

To solve these problems, the paper proposes a method using a "perfect" network model to test network security decision - making tools. This method promotes testing by providing a known and consistent standard for comparison, and allows researchers to select specific levels of error, noise, and uncertainty to evaluate the performance of network security tools under these experimental conditions.

### Summary of main problems:
1. **Inconsistent evaluation criteria**: Existing evaluation methods lack consistency and repeatability, making it difficult to accurately measure the performance of tools.
2. **Unstable opponent quality**: In different test environments, the opponents' capabilities may be different, affecting the reliability of evaluation results.
3. **Lack of controllable variables**: Existing methods are difficult to introduce controllable errors, noise, and uncertainty, and cannot comprehensively evaluate the performance of tools in complex environments.

### Solutions:
- **"Perfect" network model**: Build an ideal network model to simulate real - attack paths and defense measures, providing a stable and controllable test environment.
- **Standardized evaluation**: Through this model, the performance of tools can be evaluated in a standardized manner, ensuring that the results of each test are comparable.
- **Introduce controllable variables**: Researchers - selected errors, noise, and uncertainty can be introduced in the test to evaluate the performance of tools under different conditions.

This method not only improves the accuracy and consistency of evaluation but also provides strong support for the research, development, and improvement of network security tools.