Abstract:The complexity of ICT infrastructures is continuously increasing, presenting a formidable challenge in safeguarding them against cyber attacks. In light of escalating cyber threats and limited availability of expert resources, organizations must explore more efficient approaches to assess their resilience and undertake proactive measures. Threat modeling is an effective approach for assessing the cyber resilience of ICT systems. One method is to utilize Attack Graphs, which visually represent the steps taken by adversaries during an attack. Previously, MAL (the Meta Attack Language) was proposed, which serves as a framework for developing Domain-Specific Languages (DSLs) and generating Attack Graphs for modeled infrastructures. coreLang is a MAL-based threat modeling language that utilizes such Attack Graphs to enable attack simulations and security assessments for the generic ICT domain. Developing domain-specific languages for threat modeling and attack simulations provides a powerful approach for conducting security assessments of infrastructures. However, ensuring the correctness of these modeling languages raises a separate research question. In this study we conduct an empirical experiment aiming to falsify such a domain-specific threat modeling language. The potential inability to falsify the language through our empirical testing would lead to its corroboration, strengthening our belief in its validity within the parameters of our study. The outcomes of this approach indicated that, on average, the assessments generated by attack simulations outperformed those of human experts. Additionally, both human experts and simulations exhibited significantly superior performance compared to random guessers in their assessments. While specific human experts occasionally achieved better assessments for particular questions in the experiments, the efficiency of simulation-generated assessments surpasses that of human domain experts.

Empirical Evaluation of a Threat Modeling Language as a Cybersecurity Assessment Tool

Cyber security threat modeling based on the MITRE Enterprise ATT&CK Matrix

CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models

AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks

A Systematic Evaluation of Artificial Intelligence's Impact on the Landscape of Threat Modeling

ThreatModeling-LLM: Automating Threat Modeling using Large Language Models for Banking System

Usefulness of data flow diagrams and large language models for security threat validation: a registered report

Assessing Large Language Model’s knowledge of threat behavior in MITRE ATT&CK

CS-Eval: A Comprehensive Large Language Model Benchmark for CyberSecurity

Automated Security Assessments of Amazon Web Services Environments

Countering Autonomous Cyber Threats

ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming

On the Uses of Large Language Models to Interpret Ambiguous Cyberattack Descriptions

Evaluating Attacker Risk Behavior in an Internet of Things Ecosystem

The Best Defense is a Good Offense: Countering LLM-Powered Cyberattacks

LLMs Killed the Script Kiddie: How Agents Supported by Large Language Models Change the Landscape of Network Threat Testing

ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models

Transforming Computer Security and Public Trust Through the Exploration of Fine-Tuning Large Language Models

CySecBERT: A Domain-Adapted Language Model for the Cybersecurity Domain

Threat Modelling and Risk Analysis for Large Language Model (LLM)-Powered Applications

MetaAID 2.5: A Secure Framework for Developing Metaverse Applications via Large Language Models