Abstract:Transformer, as one of the most advanced neural network models in Natural Language Processing (NLP), exhibits diverse applications in the field of anomaly detection. To inspire research on Transformer-based anomaly detection, this review offers a fresh perspective on the concept of anomaly detection. We explore the current challenges of anomaly detection and provide detailed insights into the operating principles of Transformer and its variants in anomaly detection tasks. Additionally, we delineate various application scenarios for Transformer-based anomaly detection models and discuss the datasets and evaluation metrics employed. Furthermore, this review highlights the key challenges in Transformer-based anomaly detection research and conducts a comprehensive analysis of future research trends in this domain. The review includes an extensive compilation of over 100 core references related to Transformer-based anomaly detection. To the best of our knowledge, this is the first comprehensive review that focuses on the research related to Transformer in the context of anomaly detection. We hope that this paper can provide detailed technical information to researchers interested in Transformer-based anomaly detection tasks.

What problem does this paper attempt to address?

The paper aims to address key issues in the research and application of Transformer-based anomaly detection models. Specifically, it explores the challenges currently faced in anomaly detection and provides a detailed explanation of the working principles of Transformers and their variants in anomaly detection tasks. Additionally, the paper outlines various application scenarios of Transformer-based anomaly detection models and discusses the datasets and evaluation metrics used. Finally, the paper identifies the main challenges in Transformer-based anomaly detection research and provides a comprehensive analysis of future research trends. ### The main issues the paper attempts to address include: 1. **Concept Clarification**: The paper provides a comprehensive overview and analysis of the concept of anomaly detection, clarifying the distinctions and connections between concepts such as anomalies, outliers, OOD (out-of-distribution), and novelty in different studies. 2. **Technical Challenges**: The paper discusses the technical challenges faced in current anomaly detection tasks, particularly the issue of data imbalance and how to effectively distinguish between normal and abnormal data in high-dimensional and complex feature data. 3. **Model Mechanism**: The paper explains in detail the working mechanism of Transformers and their variants in anomaly detection tasks, including how their unique attention mechanism improves the performance of contextual feature extraction. 4. **Application Domains**: The paper summarizes the applications of Transformer-based anomaly detection models in various fields, such as system logs, industrial control systems, autonomous driving systems, medical diagnostics, spacecraft, etc. 5. **Future Trends**: The paper analyzes future trends in Transformer-based anomaly detection research, proposing open questions and research gaps, providing directions for future research. Through these contents, the paper aims to provide researchers with detailed technical information and inspire further research in the field of Transformer-based anomaly detection.

Research and application of Transformer based anomaly detection model: A literature review

Comprehensive review of Transformer‐based models in neuroscience, neurology, and psychiatry

Transformers in Machine Learning: Literature Review

Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy

Transformer based Models for Unsupervised Anomaly Segmentation in Brain MR Images

What Information Contributes to Log-based Anomaly Detection? Insights from a Configurable Transformer-Based Approach

Transformers in Natural Language Processing: A Comprehensive Review

UTRAD: Anomaly detection and localization with U-Transformer

Log-Based Anomaly Detection with Transformers Pre-Trained on Large-Scale Unlabeled Data

ITran: A novel transformer-based approach for industrial anomaly detection and localization

TransAnomaly: Video Anomaly Detection Using Video Vision Transformer

Transformer-based approaches for neuroimaging: an in-depth review of their role in classification and regression tasks

Concept Drift Adaptation for Time Series Anomaly Detection via Transformer

Advancements in Natural language Processing: An In-depth Review of Language Transformer Models

Transformer-based intelligent fault diagnosis methods of mechanical equipment: A survey

BTAD: A binary transformer deep neural network model for anomaly detection in multivariate time series data

DMU-TransNet: Dense multi-scale U-shape transformer network for anomaly detection

A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks

LGAT: A Novel Model for Multivariate Time Series Anomaly Detection with Imporved Anomaly Transformer and Learning Graph Structures

Transformer-based contrastive learning framework for image anomaly detection

A Transformer Architecture based mutual attention for Image Anomaly Detection