Abstract:The increasing availability of longitudinal data (repeated numerical observations of the same units at different times) requires the development of flexible techniques to automatically detect errors in such data. Besides standard types of errors, which can be treated with generic error correction techniques, large longitudinal datasets may present specific problems not easily traceable by the generic techniques. In particular, after applying those generic techniques, time series in the data may contain trends, natural fluctuations and possible surviving errors. To study the data evolution, one main issue is distinguishing those elusive errors from the rest, which should be kept as they are and not flattened or altered. This work responds to this need by identifying some types of elusive errors and by proposing a statistical-mathematical approach to capture their complexity that can be applied after the above generic techniques. The proposed approach is based on a system of indicators and works at the formal level by studying the differences between consecutive values of data series and the symmetries and asymmetries of these differences. It operates regardless of the specific meaning of the data and is thus applicable in a variety of contexts. We implement this approach in a relevant database of European Higher Education institutions (ETER) by analyzing two key variables: "Total academic staff" and "Total number of enrolled students", which are two of the most important variables, often used in empirical analysis as a proxy for size, and are considered by policymakers at the European level. The results are very promising.

Cleanits-MEDetect: Multiple Errors Detection for Time Series Data in Cleanits.

Cleanits

Multiscale water quality contamination events detection based on sensitive time scales reconstruction

A Data-Driven Fault Detection Framework Using Mahalanobis Distance Based Dynamic Time Warping

Water quality event detection based on Multivariate empirical mode decomposition

Data Cleaning Based on Multi-sensor Spatiotemporal Correlation

An Unsupervised Error Detection Methodology for Detecting Mislabels in Healthcare Analytics

IHCS: An Integrated Hybrid Cleaning System.

Time Series Data Cleaning under Expressive Constraints on Both Rows and Columns

Flexible Techniques to Detect Typical Hidden Errors in Large Longitudinal Datasets

Time Series Data Cleaning: From Anomaly Detection to Anomaly Repairing (Technical Report)

MSIDetector: Detecting Multi-Scenario Industrial Defects Using an Adapted Visual Foundation Model and Dual Thresholding Discriminator

Akane: Perplexity-Guided Time Series Data Cleaning

BoostClean: Automated Error Detection and Repair for Machine Learning

Online Fault Diagnosis in Industrial Processes Using Multimodel Exponential Discriminant Analysis Algorithm

A Multi-Model Exponential Discriminant Analysis Algorithm for Online Probabilistic Diagnosis of Time-Varying Faults.

Time Series Data Cleaning with Regular and Irregular Time Intervals

A Dirty Data Recognition Method for Machinery Condition Monitoring in Big Data Era.

Detection of time delays and directional interactions based on time series from complex dynamical systems.

ADET: anomaly detection in time series with linear time

Datactive: Data Fault Localization for Object Detection Systems