Abstract:Due to the data imbalance and the diversity of defects, student-teacher networks (S-T) are favored in unsupervised anomaly detection, which explores the discrepancy in feature representation derived from the knowledge distillation process to recognize anomalies. However, vanilla S-T network is not stable. Employing identical structures to construct the S-T network may weaken the representative discrepancy on anomalies. But using different structures can increase the likelihood of divergent performance on normal data. To address this problem, we propose a novel dual-student knowledge distillation (DSKD) architecture. Different from other S-T networks, we use two student networks a single pre-trained teacher network, where the students have the same scale but inverted structures. This framework can enhance the distillation effect to improve the consistency in recognition of normal data, and simultaneously introduce diversity for anomaly representation. To explore high-dimensional semantic information to capture anomaly clues, we employ two strategies. First, a pyramid matching mode is used to perform knowledge distillation on multi-scale feature maps in the intermediate layers of networks. Second, an interaction is facilitated between the two student networks through a deep feature embedding module, which is inspired by real-world group discussions. In terms of classification, we obtain pixel-wise anomaly segmentation maps by measuring the discrepancy between the output feature maps of the teacher and student networks, from which an anomaly score is computed for sample-wise determination. We evaluate DSKD on three benchmark datasets and probe the effects of internal modules through ablation experiments. The results demonstrate that DSKD can achieve exceptional performance on small models like ResNet18 and effectively improve vanilla S-T networks.

Anomaly detection based on multi-teacher knowledge distillation

Autoencoder-Like Knowledge Distillation Network for Anomaly Detection

VDKD: A ViT-Based Student-Teacher Knowledge Distillation for Multi-Texture Class Anomaly Detection

Pull & Push: Leveraging Differential Knowledge Distillation for Efficient Unsupervised Anomaly Detection and Localization

Knowledge Distillation Method for Surface Defect Detection.

Dual-student knowledge distillation for visual anomaly detection

Unsupervised anomaly detection and localization via bidirectional knowledge distillation

Research on Knowledge Distillation Algorithm of Object Detection

Advancing Pre-trained Teacher: Towards Robust Feature Discrepancy for Anomaly Detection

Anomaly Detection via Reverse Distillation from One-Class Embedding

Dual-Modeling Decouple Distillation for Unsupervised Anomaly Detection

Remembering Normality: Memory-guided Knowledge Distillation for Unsupervised Anomaly Detection

Cosine similarity knowledge distillation for surface anomaly detection

AEKD: Unsupervised auto-encoder knowledge distillation for industrial anomaly detection

Attend, Distill, Detect: Attention-aware Entropy Distillation for Anomaly Detection

Dual-Student Knowledge Distillation Networks for Unsupervised Anomaly Detection

Structural Teacher-Student Normality Learning for Multi-Class Anomaly Detection and Localization

RDMS: Reverse distillation with multiple students of different scales for anomaly detection

Enhanced multi-scale features mutual mapping fusion based on reverse knowledge distillation for industrial anomaly detection and localization

MixedTeacher : Knowledge Distillation for fast inference textural anomaly detection

Unlocking the Potential of Reverse Distillation for Anomaly Detection