Abstract:Semi-supervised anomaly detection for sensor signals is critical in ensuring system reliability in smart manufacturing. However, existing methods rely heavily on data correlation, neglecting causality and leading to potential misinterpretations due to confounding factors. Moreover, while current reinforcement learning-based methods can effectively identify known and unknown anomalies with limited labeled samples, these methods still face several challenges, such as under-utilization of priori knowledge, lack of model flexibility, and deficient reward feedback during environmental interactions. To address the above problems, this paper innovatively constructs a counterfactual causal reinforcement learning model, termed Triple-Assisted Causal Reinforcement Learning Anomaly Detector (Tri-CRLAD). The model leverages causal inference to extract the intrinsic causal feature in data, enhancing the agent's utilization of prior knowledge and improving its generalization capability. In addition, Tri-CRLAD features a triple decision support mechanism, including a sampling strategy based on historical similarity, an adaptive threshold smoothing adjustment strategy, and an adaptive decision reward mechanism. These mechanisms further enhance the flexibility and generalization ability of the model, enabling it to effectively respond to various complex and dynamically changing environments. Experimental results across seven diverse sensor signal datasets demonstrate that Tri-CRLAD outperforms nine state-of-the-art baseline methods. Notably, Tri-CRLAD achieves up to a 23\% improvement in anomaly detection stability with minimal known anomaly samples, highlighting its potential in semi-supervised anomaly detection scenarios. Our code is available at

What problem does this paper attempt to address?

The paper mainly addresses the issue of semi-supervised anomaly detection in sensor signals within an intelligent manufacturing environment and proposes a method based on adaptive reinforcement learning and causal inference—Tri-CRLAD (Tri-auxiliary Causal Reinforcement Learning Anomaly Detector). ### Problems the Paper Attempts to Solve 1. **Causal Relationships Ignored**: Existing methods overly rely on correlations between data while ignoring causal relationships, which may lead to misjudgments due to confounding factors. 2. **Challenges with Limited Labeled Samples**: Current reinforcement learning-based methods can effectively identify known and unknown anomalies with limited labeled samples but still face issues such as insufficient use of prior knowledge, lack of model flexibility, and inadequate reward feedback during environmental interactions. 3. **Fixed Threshold Limits Model Generalization**: Existing methods typically use fixed thresholds to determine anomalies, which limits the model's generalization ability and increases the complexity of parameter tuning. 4. **Reward Mechanism Limitations**: The current reward mechanisms are relatively simple and fixed, unable to dynamically adjust according to the actual sensor signal context, leading to inefficiency during the exploration phase. ### Proposed Method To address the above issues, the paper proposes the Tri-CRLAD model, which has the following features: - **Causal Reinforcement Learning**: By constructing a counterfactual causal reinforcement learning model, it extracts intrinsic causal features from the data, thereby enhancing the model's use of prior knowledge and its generalization ability. - **Triple Decision Support Mechanism**: - Sampling strategy based on historical similarity: Ensures the model can explore data points more broadly, reducing the possibility of repeated sampling. - Adaptive threshold smoothing adjustment strategy: Overcomes the limitations brought by fixed thresholds, improving the model's flexibility. - Adaptive decision reward mechanism: Dynamically adjusts reward feedback according to environmental changes, enhancing the model's learning efficiency. ### Main Contributions 1. **Innovative Combination of Causal Inference and Reinforcement Learning**: By identifying intrinsic causal features in the data through counterfactual causal inference, the model's use of prior knowledge and generalization ability are improved. 2. **Comprehensive Triple Decision Support Mechanism**: Including a sampling strategy based on historical similarity, an adaptive threshold smoothing adjustment strategy, and an adaptive decision reward mechanism, significantly enhancing the training efficiency and generalization ability of Tri-CRLAD. 3. **Experimental Results**: Extensive experiments on multiple sensor signal datasets show that Tri-CRLAD performs excellently in semi-supervised anomaly detection, outperforming nine state-of-the-art baseline methods. Notably, its anomaly detection stability improves by up to 23% in scenarios with only a small number of known anomaly samples.

Semi-supervised Anomaly Detection via Adaptive Reinforcement Learning-Enabled Method with Causal Inference for Sensor Signals

Cognitive Sensing: Adaptive Anomalies Detection with Deep Networks

Deep Anomaly Detection and Search via Reinforcement Learning

Deep Anomaly Detection and Search Via Reinforcement Learning (student Abstract)

Toward Deep Supervised Anomaly Detection: Reinforcement Learning from Partially Labeled Anomaly Data

Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals

Dual-input anomaly detection method based on deep reinforcement learning

Deep Anomaly Detection Via Active Anomaly Search.

Application of Improved Asynchronous Advantage Actor Critic Reinforcement Learning Model on Anomaly Detection

Unraveling the "Anomaly" in Time Series Anomaly Detection: A Self-supervised Tri-domain Solution

Context-aware Feature Reconstruction for Class-Incremental Anomaly Detection and Localization

A Causal Inference Look at Unsupervised Video Anomaly Detection

Comparative Study on Semi-supervised Learning Applied for Anomaly Detection in Hydraulic Condition Monitoring System

Rectifying inaccurate unsupervised learning for robust time series anomaly detection

Explainable Online Unsupervised Anomaly Detection for Cyber-Physical Systems via Causal Discovery from Time Series

Unsupervised Continual Anomaly Detection with Contrastively-learned Prompt

Reinforcement Learning-based Anomaly Detection for PHM applications

Self-supervised Feature Adaptation for 3D Industrial Anomaly Detection

Towards Experienced Anomaly Detector Through Reinforcement Learning.

Learning Unified Reference Representation for Unsupervised Multi-class Anomaly Detection

Learning Discriminative Features for Semi-Supervised Anomaly Detection.