Abstract:The ever-evolving landscape of attacks, coupled with the growing complexity of ICT systems, makes crafting anomaly-based intrusion detectors (ID) and error detectors (ED) a difficult task: they must accurately detect attacks, and they should promptly perform detections. Although improving and comparing the detection capability is the focus of most research works, the timeliness of the detection is less considered and often insufficiently evaluated or discussed. In this paper, we argue the relevance of measuring the temporal latency of attacks and errors, and we propose an evaluation approach for detectors to ensure a pragmatic trade-off between correct and in-time detection. Briefly, the approach relates the false positive rate with the temporal latency of attacks and errors, and this ultimately leads to guidelines for configuring a detector. We apply our approach by evaluating different ED and ID solutions in two industrial cases: i) an embedded railway on-board system that optimizes public mobility, and ii) an edge device for the Industrial Internet of Things. Our results show that considering latency in addition to traditional metrics like the false positive rate, precision, and coverage gives an additional fundamental perspective on the actual performance of the detector and should be considered when assessing and configuring anomaly detectors.

What problem does this paper attempt to address?

The main problems that this paper attempts to solve are: **When evaluating anomaly detectors (including intrusion detection systems and error detection systems), most of the existing research focuses on the accuracy of detection and ignores the important factor of detection latency**. Specifically: 1. **Neglect of Detection Latency**: - Most of the current research work mainly focuses on improving and comparing detection capabilities, but insufficiently considers the timeliness of detection and fails to fully evaluate or discuss it. - Detection latency refers to the time interval from the occurrence of an attack or error to its detection, and this factor is crucial for ensuring the security and reliability of the system. 2. **Limitations of Existing Evaluation Metrics**: - Commonly used evaluation metrics such as accuracy, F1 - score, precision, and recall can measure the correctness of classification, but cannot reflect the timeliness of detection. - These metrics may lead to a one - sided understanding of the detector's performance, especially in an operating environment where delayed detection may lead to serious consequences. 3. **Problems in Dataset Construction**: - The training dataset usually contains data of long - term normal operations and multiple injected attacks, which makes the process of the system transitioning from a normal state to an abnormal state less reproduced in the data, thus affecting the detector's ability to learn such dynamic changes. - The time - order information of data points is not fully utilized, resulting in difficulty in capturing context and group anomalies. To solve these problems, the paper proposes a new evaluation method that takes detection latency into account and verifies the effectiveness of this method through experiments. Specifically, the goals of the paper are: - **Introduce detection latency as an evaluation metric** to ensure that detection is not only accurate but also timely. - **Establish a trade - off model** to find the optimal balance between correct detection and timely detection. - **Verify the effectiveness and practicality of the proposed evaluation method through two industrial cases (embedded railway on - board systems and industrial Internet of Things edge devices)**. Through these efforts, the paper aims to provide a more comprehensive method for evaluating and configuring anomaly detectors, thereby improving their performance and reliability in practical applications.

Detection Latencies of Anomaly Detectors: An Overlooked Perspective ?

Adversarial Attacks and Mitigation for Anomaly Detectors of Cyber-Physical Systems

Packet Payload Based Anomalous Network Intrusion Detection

A Real Time Deep Learning based Approach for Detecting Network Attacks

A Framework for the Systematic Assessment of Anomaly Detectors in Time-Sensitive Automotive Networks

Anomaly-based network intrusion detection: Techniques, systems and challenges

Detection of Hidden Attacks on Cyber-Physical Systems from Serial Magnitude and Sign Randomness Inconsistencies

An overview of anomaly detection techniques: Existing solutions and latest technological trends

Effective Anomaly Detection in Smart Home by Integrating Event Time Intervals

Analyzing and Improving Performance of a Class of Anomaly-based Intrusion Detectors

Statistical Evaluation of Anomaly Detectors for Sequences

Design and Experimental Assessment of Real-Time Anomaly Detection Techniques for Automotive Cybersecurity

Anomaly Detection in Railway Sensor Data Environments: State-of-the-Art Methods and Empirical Performance Evaluation

Learning-Based Simultaneous Detection and Characterization of Time Delay Attack in Cyber-Physical Systems

Using Ensemble Learning for Anomaly Detection in Cyber–Physical Systems

Should I Raise The Red Flag? A comprehensive survey of anomaly scoring methods toward mitigating false alarms

Are Concept Drift Detectors Reliable Alarming Systems? -- A Comparative Study

A Pragmatical Approach to Anomaly Detection Evaluation in Edge Cloud Systems

An Ontological Approach to the Detection of Anomalies in Vehicular Ad Hoc Networks

PATE: Proximity-Aware Time series anomaly Evaluation