Ollabench: Evaluating LLMs' Reasoning for Human-centric Interdependent Cybersecurity

Tam n. Nguyen
2024-06-11
Abstract:Large Language Models (LLMs) have the potential to enhance Agent-Based Modeling by better representing complex interdependent cybersecurity systems, improving cybersecurity threat modeling and risk management. However, evaluating LLMs in this context is crucial for legal compliance and effective application development. Existing LLM evaluation frameworks often overlook the human factor and cognitive computing capabilities essential for interdependent cybersecurity. To address this gap, I propose OllaBench, a novel evaluation framework that assesses LLMs' accuracy, wastefulness, and consistency in answering scenario-based information security compliance and non-compliance questions. OllaBench is built on a foundation of 24 cognitive behavioral theories and empirical evidence from 38 peer-reviewed papers. OllaBench was used to evaluate 21 LLMs, including both open-weight and commercial models from OpenAI, Anthropic, Google, Microsoft, Meta and so on. The results reveal that while commercial LLMs have the highest overall accuracy scores, there is significant room for improvement. Smaller low-resolution open-weight LLMs are not far behind in performance, and there are significant differences in token efficiency and consistency among the evaluated models. OllaBench provides a user-friendly interface and supports a wide range of LLM platforms, making it a valuable tool for researchers and solution developers in the field of human-centric interdependent cybersecurity and beyond.
Cryptography and Security,Artificial Intelligence,Human-Computer Interaction
What problem does this paper attempt to address?
This paper proposes a solution to the challenges in evaluating Human-Centric Interdependent Network Security (HCINS) using Large Language Models (LLMs). Existing LLM evaluation frameworks often disregard human factors and cognitive computing capabilities, which are crucial in handling complex network security systems. The authors introduce OllaBench, a novel evaluation framework for assessing the accuracy, redundancy, and consistency of LLMs in addressing scenario-based information security compliance and non-compliance questions. OllaBench is constructed based on 24 cognitive behavior theories and empirical evidence from 38 peer-reviewed papers, and it evaluates 21 LLMs. OllaBench supports multiple LLM platforms through a user-friendly interface, aiming to assist researchers and solution developers in conducting more effective research and development in the HCINS field and other related domains. The study reveals that while commercial LLMs perform the best overall in terms of accuracy, there is still room for improvement, and smaller, low-resolution open-source weight LLMs are not far behind in performance. Additionally, significant differences exist among different models in terms of token efficiency and consistency. The paper points out that as humans play a significant role in network security issues, reinforcing human factors is an important strategy for enhancing network security risk management effectiveness. OllaBench provides an evaluation approach that helps ensure the security, effectiveness, and compliance of LLMs when applied to network security.