Abstract:Prognostics and Health Management (PHM) is an essential requirement for engineering assets. Its processing strategies include modules for the detection, diagnostics and prognostics of known fault conditions. However, during operation, there are always fault conditions that were not anticipated. These events manifest as anomalies and could potentially be catastrophic with the loss of the asset. Anomalies can indicate an impending fault condition, therefore, the automatic identification of anomalies can lead to solving reliability problems that might manifest because of complexities arising from the operating environment and component degradation. Data-driven approaches have gained increasing popularity as a comprehensive anomaly detection method whenever data on nominal and fault conditions is available. However, many supervised learning techniques often face problems whenever models are trained from the limited set of partially labelled anomalies, whilst the rest of the dataset is left unlabelled. An alternative is to use unsupervised learning techniques, that are supposed to obviate stipulating the performance of the anomaly detector. But these still often produce many false positives because of the lack of prior knowledge of true anomalies. Considering this, this article investigates the use of a Reinforcement Learning (RL)-based approach to address the problem of unknown classes of anomalies that might lie beyond the scope of the initially trained model. A Q-learning method is used to exploit the existing data model whilst exploring new classes to improve classification accuracy and optimise decision making. This makes it of significant practical benefit, as anomalies can be unpredictable in form and usually evolve over time. In particular, a deep network-based anomaly detector agent is used to initially learn the action-value function (i.e., the Q-value function) from the limited labelled data. An environment is created for the agent to actively interact not only with the labelled anomalies but also to explore rare and novel unlabelled anomalies that might lie beyond the scope of the initially trained model. A reward function is defined based on the sparse normative content, which stipulates when the agent detects the anomaly state. However, the robustness of this method is still an open question as it simply shifts the anomaly detection responsibility onto the reward function being used. This shows the strong dependence on how the problem state-action space is defined for these methods to perform well.

Reinforcement Learning-based Anomaly Detection for PHM applications

Deep Reinforcement Learning for Anomaly Detection: A Systematic Review

Toward Deep Supervised Anomaly Detection: Reinforcement Learning from Partially Labeled Anomaly Data

Deep reinforcement learning for data-efficient weakly supervised business process anomaly detection

Application of Improved Asynchronous Advantage Actor Critic Reinforcement Learning Model on Anomaly Detection

Towards Experienced Anomaly Detector Through Reinforcement Learning.

Deep Anomaly Detection and Search via Reinforcement Learning

A Distance-based Anomaly Detection Framework for Deep Reinforcement Learning

Agent-based dynamic thresholding for adaptive anomaly detection using reinforcement learning

Dual-input anomaly detection method based on deep reinforcement learning

Deep Anomaly Detection and Search Via Reinforcement Learning (student Abstract)

Semi-supervised Anomaly Detection via Adaptive Reinforcement Learning-Enabled Method with Causal Inference for Sensor Signals

Development of data anomaly classification for structural health monitoring based on iterative trimmed loss minimization and human-in-the-loop learning

Real-Time Predictive Maintenance using Autoencoder Reconstruction and Anomaly Detection

Using Ensemble Learning for Anomaly Detection in Cyber–Physical Systems

Machine Learning-Based Anomaly Detection Using K-mean Array and Sequential Minimal Optimization

Anomaly Detection via Learning-Based Sequential Controlled Sensing

Unsupervised novelty detection for time series using a deep learning approach

Deep Anomaly Detection Via Active Anomaly Search.

A Deep Learning Approach to Anomaly Sequence Detection for High-Resolution Monitoring of Power Systems

Classification-Based Self-Supervised Learning For Anomaly Detection