Abstract:Novelty detection in discrete sequences is a challenging task, since deviations from the process generating the normal data are often small or intentionally hidden. Novelties can be detected by modeling normal sequences and measuring the deviations of a new sequence from the model predictions. However, in many applications data is generated by several distinct processes so that models trained on all the data tend to over-generalize and novelties remain undetected. We propose to approach this challenge through decomposition: by clustering the data we break down the problem, obtaining simpler modeling task in each cluster which can be modeled more accurately. However, this comes at a trade-off, since the amount of training data per cluster is reduced. This is a particular problem for discrete sequences where state-of-the-art models are data-hungry. The success of this approach thus depends on the quality of the clustering, i.e., whether the individual learning problems are sufficiently simpler than the joint problem. While clustering discrete sequences automatically is a challenging and domain-specific task, it is often easy for human domain experts, given the right tools. In this paper, we adapt a state-of-the-art visual analytics tool for discrete sequence clustering to obtain informed clusters from domain experts and use LSTMs to model each cluster individually. Our extensive empirical evaluation indicates that this informed clustering outperforms automatic ones and that our approach outperforms state-of-the-art novelty detection methods for discrete sequences in three real-world application scenarios. In particular, decomposition outperforms a global model despite less training data on each individual cluster.

The empirical impact of the nature of novelty detection

The nature of novelty detection

Variation of the glass transition temperature with rigidity and chemical composition

An Improved System for Sentence-level Novelty Detection in Textual Streams

A Systematic Literature Review of Novelty Detection in Data Streams: Challenges and Opportunities

NovAScore: A New Automated Metric for Evaluating Document Level Novelty

A comparative evaluation of novelty detection algorithms for discrete sequences

Efficient Novelty Search Through Deep Reinforcement Learning

THU TREC 2002: Novelty Track Experiments.

Novelty and Topicality in Interactive Information Retrieval

NovelCraft: A Dataset for Novelty Detection and Discovery in Open Worlds

How effective is query expansion for finding novel information?

Effect of the Topic-Combination Novelty on the Disruption and Impact of Scientific Articles: Evidence from PubMed

Improved Feature Selection and Redundance Computing - THUIR at TREC 2004 Novelty Track.

L1 norm based KPCA for novelty detection

Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications

A Content-Based Novelty Measure for Scholarly Publications: A Proof of Concept

Novelty Detection in Sequential Data by Informed Clustering and Modeling

Text Classification with Novelty Detection

Continual Novelty Detection

Predictive Effects of Novelty Measured by Temporal Embeddings on the Growth of Scientific Literature