Scalable and automated Evaluation of Blue Team cyber posture in Cyber Ranges

Federica Bianchi,Enrico Bassetti,Angelo Spognardi

DOI: https://doi.org/10.1145/3605098.3636154

2023-12-29

Abstract:Cyber ranges are virtual training ranges that have emerged as indispensable environments for conducting secure exercises and simulating real or hypothetical scenarios. These complex computational infrastructures enable the simulation of attacks, facilitating the evaluation of defense tools and methodologies and developing novel countermeasures against threats. One of the main challenges of cyber range scalability is the exercise evaluation that often requires the manual intervention of human operators, the White team. This paper proposes a novel approach that uses Blue and Red team reports and well-known databases to automate the evaluation and assessment of the exercise outcomes, overcoming the limitations of existing assessment models. Our proposal encompasses evaluating various aspects and metrics, explicitly emphasizing Blue Teams' actions and strategies and allowing the automated generation of their cyber posture.

Cryptography and Security

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **How to automate the evaluation and analysis of the performance of the Blue Team in cyber ranges to overcome the limitations of existing manual evaluation methods**.

### Specific problem description:

1. **Limitations of manual evaluation**:
- Existing evaluation methods rely on service indicators and manual scoring by human experts. This method is time - consuming, error - prone, and unable to provide timely feedback on the Blue Team's response.
- The manual evaluation process often requires a great deal of participation from the White Team, which increases the workload and may lead to subjectivity and inconsistency in the evaluation results.

2. **Complexity of the evaluation process**:
- Evaluating the performance of the Blue Team requires considering multiple aspects, including attack management, understanding of attack strategies, mastery of (sub - ) techniques, accuracy of identification techniques, response speed, etc.
- Traditional evaluation methods are difficult to comprehensively cover these aspects, resulting in evaluation results that are not comprehensive and objective enough.

3. **Scaling challenges**:
- As the scale of cyber security exercises expands, manual evaluation methods become increasingly infeasible. Existing evaluation models cannot effectively cope with large - scale, multi - team parallel exercise scenarios.

### Solutions proposed in the paper:

To overcome the above problems, this paper proposes a new automated evaluation framework, which specifically includes the following aspects:

1. **Automatically generate evaluation reports**:
- Use the reports of the Red Team and the Blue Team and known security databases (such as MITRE ATT&CK and CAPEC) to automatically generate the Reference Graph and Response Graph required for evaluation.

2. **Graph - structure - based evaluation method**:
- Use the ADTree (tree - like structure) to model attack and defense scenarios, and evaluate the performance of the Blue Team by comparing the Reference Graph and the Response Graph.
- Through the Breadth - First Search (BFS) algorithm, automatically assign weights to the correctly identified nodes of the Blue Team and remove unmatched nodes.

3. **Multi - dimensional scoring system**:
- Define multiple intermediate scores (such as understanding score, defense score, implementation score, and response score) to comprehensively evaluate the performance of the Blue Team in different aspects.
- According to different evaluation objectives, weights can be assigned to each intermediate score, and finally a comprehensive Final Score can be calculated.

4. **Automation and scalability**:
- This framework aims to provide an efficient, objective, and scalable evaluation method that can evaluate the performance of multiple Blue Teams simultaneously and reduce the need for human intervention.

Through these improvements, this paper hopes to significantly improve the efficiency and accuracy of Blue Team evaluation in cyber security exercises, thereby helping organizations better enhance their cyber security defense capabilities.