Abstract:Metrics and frameworks to quantifiably assess security measures have arisen from needs of three distinct research communities - statistical measures from the intrusion detection and prevention literature, evaluation of cyber exercises, e.g.,red-team and capture-the-flag competitions, and economic analyses addressing cost-versus-security tradeoffs. In this paper we provide two primary contributions to the security evaluation literature - a representative survey, and a novel framework for evaluating security that is flexible, applicable to all three use cases, and readily interpretable. In our survey of the literature we identify the distinct themes from each community's evaluation procedures side by side and flesh out the drawbacks and benefits of each. The evaluation framework we propose includes comprehensively modeling the resource, labor, and attack costs in dollars incurred based on expected resource usage, accuracy metrics, and time. This framework provides a unified approach in that it incorporates the accuracy and performance metrics, which dominate intrusion detection evaluation, the time to detection and impact to data and resources of an attack, favored by educational competitions' metrics, and the monetary cost of many essential security components used in financial analysis. Moreover, it is flexible enough to accommodate each use case, easily interpretable and comparable, and comprehensive in terms of costs <a class="link-external link-http" href="http://considered.Finally" rel="external noopener nofollow">this http URL</a>, we provide two examples of the framework applied to real-world use cases. Overall, we provide a survey and a grounded, flexible framework with multiple concrete examples for evaluating security which can address the needs of three currently distinct communities.

From Lemons to Peaches: Improving Security ROI through Security Chaos Engineering

Rethinking Security Incident Response: The Integration of Agile Principles

Finding Effective Security Strategies through Reinforcement Learning and Self-Play

Enhancing DevSecOps practice with Large Language Models and Security Chaos Engineering

Towards a More Systematic Approach to Secure Systems Design and Analysis

Computing Optimal Security Strategies for Interdependent Assets

Economics of Information Security Investment Integrated with IDS and Attacker’s Behavior

Security Analysis of Cyber-Physical Systems Using Reinforcement Learning

Quantifiable & Comparable Evaluations of Cyber Defensive Capabilities: A Survey & Novel, Unified Approach

Fast, Cheap and Good: Lightweight Methods Are Undervalued

Security in IS and social engineering -- an overview and state of the art

Cybersecurity Landscape on Remote State Estimation: A Comprehensive Review

Chaos Engineering for Enhanced Resilience of Cyber-Physical Systems

Getting Users Smart Quick about Security: Results from 90 Minutes of Using a Persuasive Toolkit for Facilitating Information Security Problem Solving by Non-Professionals

Evaluating Attacker Risk Behavior in an Internet of Things Ecosystem

Optimizing Investments in Cyber Hygiene for Protecting Healthcare Users

Optimal security hardening over a probabilistic attack graph: a case study of an industrial control system using the CySecTool tool

Security Requirements Engineering in the Wild: A Survey of Common Practices

Chaos Engineering

Better Call Saltzer \& Schroeder: A Retrospective Security Analysis of SolarWinds \& Log4j

Attack Surface Metrics and Privilege-based Reduction Strategies for Cyber-Physical Systems