Abstract:Log anomaly detection is an efficient method to manage modern large-scale Internet of Things (IoT) systems. More and more works start to apply natural language processing (NLP) methods, and in particular word2vec, in the log feature extraction. Word2vec can extract the relevance between words and vectorize the words. However, the computing cost of training word2vec is high. Anomalies in logs are dependent on not only an individual log message but also on the log message sequence. Therefore, the vector of words from word2vec can not be used directly, which needs to be transformed into the vector of log events and further transformed into the vector of log sequences. To reduce computational cost and avoid multiple transformations, in this paper, we propose an offline feature extraction model, named LogEvent2vec, which takes the log event as input of word2vec to extract the relevance between log events and vectorize log events directly. LogEvent2vec can work with any coordinate transformation methods and anomaly detection models. After getting the log event vector, we transform log event vector to log sequence vector by bary or tf-idf and three kinds of supervised models (Random Forests, Naive Bayes, and Neural Networks) are trained to detect the anomalies. We have conducted extensive experiments on a real public log dataset from BlueGene/L (BGL). The experimental results demonstrate that LogEvent2vec can significantly reduce computational time by 30 times and improve accuracy, comparing with word2vec. LogEvent2vec with bary and Random Forest can achieve the best F1-score and LogEvent2vec with tf-idf and Naive Bayes needs the least computational time.

Micro2vec: Anomaly detection in microservices systems by mining numeric representations of computer logs

BertHTLG: Graph-Based Microservice Anomaly Detection Through Sentence-Bert Enhancement.

MoniLog: An Automated Log-Based Anomaly Detection System for Cloud Computing Infrastructures

LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things

Unsupervised Anomaly Detection on Microservice Traces through Graph VAE

Anomaly detection in microservice environments using distributed tracing data analysis and NLP

Experience Report: Log Mining Using Natural Language Processing and Application to Anomaly Detection

LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs

Anomaly Detection in Microservice-Based Systems

An Intelligent Anomaly Detection Scheme for Micro-Services Architectures with Temporal and Spatial Data Analysis.

Benchmarking Micro2Micro transformation: an approach with GNN and VAE

Root-Cause Metric Location for Microservice Systems Via Log Anomaly Detection

Look Deep into the Microservice System Anomaly Through Very Sparse Logs

ServiceAnomaly: An anomaly detection approach in microservices using distributed traces and profiling metrics

DeepTraLog: Trace-Log Combined Microservice Anomaly Detection through Graph-based Deep Learning

A COMPARATIVE ANALYSIS OF ANOMALY DETECTION FROM MICROSERVICE GENERATED UNSTRUCTURED LOGS

Robust and Transferable Anomaly Detection in Log Data using Pre-Trained Language Models

Log-based Anomaly Detection Without Log Parsing

Log-based Anomaly Detection based on EVT Theory with feedback

Detecting log anomaly using subword attention encoder and probabilistic feature selection