Abstract:Prognostics and Health Management (PHM) is an essential requirement for engineering assets. Its processing strategies include modules for the detection, diagnostics and prognostics of known fault conditions. However, during operation, there are always fault conditions that were not anticipated. These events manifest as anomalies and could potentially be catastrophic with the loss of the asset. Anomalies can indicate an impending fault condition, therefore, the automatic identification of anomalies can lead to solving reliability problems that might manifest because of complexities arising from the operating environment and component degradation. Data-driven approaches have gained increasing popularity as a comprehensive anomaly detection method whenever data on nominal and fault conditions is available. However, many supervised learning techniques often face problems whenever models are trained from the limited set of partially labelled anomalies, whilst the rest of the dataset is left unlabelled. An alternative is to use unsupervised learning techniques, that are supposed to obviate stipulating the performance of the anomaly detector. But these still often produce many false positives because of the lack of prior knowledge of true anomalies. Considering this, this article investigates the use of a Reinforcement Learning (RL)-based approach to address the problem of unknown classes of anomalies that might lie beyond the scope of the initially trained model. A Q-learning method is used to exploit the existing data model whilst exploring new classes to improve classification accuracy and optimise decision making. This makes it of significant practical benefit, as anomalies can be unpredictable in form and usually evolve over time. In particular, a deep network-based anomaly detector agent is used to initially learn the action-value function (i.e., the Q-value function) from the limited labelled data. An environment is created for the agent to actively interact not only with the labelled anomalies but also to explore rare and novel unlabelled anomalies that might lie beyond the scope of the initially trained model. A reward function is defined based on the sparse normative content, which stipulates when the agent detects the anomaly state. However, the robustness of this method is still an open question as it simply shifts the anomaly detection responsibility onto the reward function being used. This shows the strong dependence on how the problem state-action space is defined for these methods to perform well.

Towards Anomaly Detection in Reinforcement Learning

Deep Anomaly Detection and Search via Reinforcement Learning

A Distance-based Anomaly Detection Framework for Deep Reinforcement Learning

A Deep Reinforcement Learning Method for Accurate and Efficient Anomaly Detection

Deep Anomaly Detection and Search Via Reinforcement Learning (student Abstract)

Deep Reinforcement Learning for Anomaly Detection: A Systematic Review

Effective Anomaly Detection Based on Reinforcement Learning in Network Traffic Data.

Rethinking Out-of-Distribution Detection for Reinforcement Learning: Advancing Methods for Evaluation and Detection

Towards Experienced Anomaly Detector Through Reinforcement Learning.

Application of Improved Asynchronous Advantage Actor Critic Reinforcement Learning Model on Anomaly Detection

Policy-based Reinforcement Learning for Time Series Anomaly Detection.

Reinforcement Learning-based Anomaly Detection for PHM applications

An Unsupervised Anomaly Detection in Electricity Consumption Using Reinforcement Learning and Time Series Forest Based Framework

Agent-based dynamic thresholding for adaptive anomaly detection using reinforcement learning

Deep Anomaly Detection Via Active Anomaly Search.

Spiking Reinforcement Learning for Weakly-Supervised Anomaly Detection

OIL-AD: An Anomaly Detection Framework for Sequential Decision Sequences

Toward Deep Supervised Anomaly Detection: Reinforcement Learning from Partially Labeled Anomaly Data

AD-LLM: Benchmarking Large Language Models for Anomaly Detection

AD-MERCS: Modeling Normality and Abnormality in Unsupervised Anomaly Detection

ALAD: A New Unsupervised Time Series Anomaly Detection Paradigm Based on Activation Learning