Abstract:River water-quality monitoring is increasingly conducted using automated in situ sensors, enabling timelier identification of unexpected values. However, anomalies caused by technical issues confound these data, while the volume and velocity of data prevent manual detection. We present a framework for automated anomaly detection in high-frequency water-quality data from in situ sensors, using turbidity, conductivity and river level data. After identifying end-user needs and defining anomalies, we ranked their importance and selected suitable detection methods. High priority anomalies included sudden isolated spikes and level shifts, most of which were classified correctly by regression-based methods such as autoregressive integrated moving average models. However, using other water-quality variables as covariates reduced performance due to complex relationships among variables. Classification of drift and periods of anomalously low or high variability improved when we applied replaced anomalous measurements with forecasts, but this inflated false positive rates. Feature-based methods also performed well on high priority anomalies, but were also less proficient at detecting lower priority anomalies, resulting in high false negative rates. Unlike regression-based methods, all feature-based methods produced low false positive rates, but did not and require training or optimization. Rule-based methods successfully detected impossible values and missing observations. Thus, we recommend using a combination of methods to improve anomaly detection performance, whilst minimizing false detection rates. Furthermore, our framework emphasizes the importance of communication between end-users and analysts for optimal outcomes with respect to both detection performance and end-user needs. Our framework is applicable to other types of high frequency time-series data and anomaly detection applications.

Unsupervised Bayesian Nonparametric Approach with Incremental Similarity Tracking of Unlabeled Water Demand Time Series for Anomaly Detection

An Unsupervised Framework for Anomaly Detection in a Water Treatment System

Dealing with Data Missing and Outlier to Calibrate Nodal Water Demands in Water Distribution Systems

Numerical Approach for Water Distribution System Model Calibration Through Incorporation of Multiple Stochastic Prior Distributions.

Leakage Localization Using Pressure Sensors and Spatial Clustering in Water Distribution Systems

A Noise Adaptive Approach for Nodal Water Demand Estimation in Water Distribution Systems

Unsupervised Anomaly Detection in Spatio‐Temporal Stream Network Sensor Data

Contamination Event Detection Method Using Multi-Stations Temporal-Spatial Information Based on Bayesian Network in Water Distribution Systems

Contamination Event Detection Method Based On The Longest Common Subsequence Analysis Using Multiple Water Quality Parameters

Anomaly Detection Using a Sliding Window Technique and Data Imputation with Machine Learning for Hydrological Time Series

Unsupervised Real Time and Early Anomalies Detection Method for Sewer Networks Systems

Real-Time Burst Detection in District Metering Areas in Water Distribution System Based on Patterns of Water Demand with Supervised Learning

Multi-parameters fusion algorithm for detecting anomalous water quality

Multiscale water quality contamination events detection based on sensitive time scales reconstruction

Detecting urban water consumption patterns: a time-series clustering approach

In-Situ Detection of Water Quality Anomaly with UV/Vis Spectrum Based on Supervised Learning

Uncovering urban water consumption patterns through time series clustering and entropy analysis

A simple method for unsupervised anomaly detection: An application to Web time series data

Short-term prediction of chaotic hourly water consumption based on united time series

A framework for automated anomaly detection in high frequency water-quality data from in situ sensors

Burst Detection by Analyzing Shape Similarity of Time Series Subsequences in District Metering Areas