Abstract:This paper is motivated by the task of detecting anomalies in networks of financial transactions, with accounts as nodes and a directed weighted edge between two nodes denoting a money transfer. The weight of the edge is the transaction amount. Examples of anomalies in networks include long paths of large transaction amounts, rings of large payments, and cliques of accounts. There are many methods available which detect such specific structures in networks. Here we introduce a method which is able to detect previously unspecified anomalies in networks. The method is based on a combination of features from network comparison and spectral analysis as well as local statistics, yielding 140 main features. We then use a simple feature sum method, as well as a random forest method, in order to classify nodes as normal or anomalous. We test the method first on synthetic networks which we generated, and second on a set of synthetic networks which were generated without the methods team having access to the ground truth. The first set of synthetic networks was split in a training set of 70 percent of the networks, and a test set of 30 percent of the networks. The resulting classifier was then applied to the second set of synthetic networks. We compare our method with Oddball, a widely used method for anomaly detection in networks, as well as to random classification. While Oddball outperforms random classification, both our feature sum method and our random forest method outperform Oddball. On the test set, the random forest outperforms feature sum, whereas on the second synthetic data set, initially feature sum tends to pick up more anomalies than random forest, with this behaviour reversing for lower-scoring anomalies. In all cases, the top 2 percent of flagged anomalies contained on average over 90 percent of the planted anomalies.

Anomaly Detection for Network Connection Logs

Anomaly detection system for network transport with machine learning approach

Anomaly Detection of Large Scale Network Based on Data Streams

Fast Memory-efficient Anomaly Detection in Streaming Heterogeneous Graphs

Big-data Analysis of Multi-Source Logs for Anomaly Detection on Network-Based System.

An Integrated Method for Anomaly Detection From Massive System Logs.

Anomaly Detection in Cyber Security with Graph-Based LSTM in Log Analysis

Log-based Anomaly Detection of Enterprise Software: An Empirical Study

MoniLog: An Automated Log-Based Anomaly Detection System for Cloud Computing Infrastructures

Anomaly Detection in Networks with Application to Financial Transaction Networks

A Big Data Architecture for Log Data Storage and Analysis

Extending Isolation Forest for Anomaly Detection in Big Data via K-Means

LogSHIELD: A Graph-based Real-time Anomaly Detection Framework using Frequency Analysis

Anomaly Detections in Internet traffic Using Empirical Measures

Anomaly detection in the CERN cloud infrastructure

Anomaly Intrusion Detection Based on Data Stream

Unsupervised real-time anomaly detection for streaming data

Development of Anomaly Detection System Based on Distributed Log Tracing

LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs

Develop End-to-End Anomaly Detection System

Anomaly detection in wide area network mesh using two machine learning anomaly detection algorithms