Abstract:The increasing availability of longitudinal data (repeated numerical observations of the same units at different times) requires the development of flexible techniques to automatically detect errors in such data. Besides standard types of errors, which can be treated with generic error correction techniques, large longitudinal datasets may present specific problems not easily traceable by the generic techniques. In particular, after applying those generic techniques, time series in the data may contain trends, natural fluctuations and possible surviving errors. To study the data evolution, one main issue is distinguishing those elusive errors from the rest, which should be kept as they are and not flattened or altered. This work responds to this need by identifying some types of elusive errors and by proposing a statistical-mathematical approach to capture their complexity that can be applied after the above generic techniques. The proposed approach is based on a system of indicators and works at the formal level by studying the differences between consecutive values of data series and the symmetries and asymmetries of these differences. It operates regardless of the specific meaning of the data and is thus applicable in a variety of contexts. We implement this approach in a relevant database of European Higher Education institutions (ETER) by analyzing two key variables: "Total academic staff" and "Total number of enrolled students", which are two of the most important variables, often used in empirical analysis as a proxy for size, and are considered by policymakers at the European level. The results are very promising.

Detecting Errors in a Numerical Response via any Regression Model

Beyond the Norms: Detecting Prediction Errors in Regression Models

A method of MPC model error detection

Censored Regression with Noisy Input

Recursive Least Squares for Censored Regression

Correntropy based data reconciliation and gross error detection for bilinear systems

A Fuzzy-Statistical Tolerance Interval from Residuals of Crisp Linear Regression Models

Model‐Checking Techniques Based on Cumulative Residuals

THE ACCEPTABILITY OF REGRESSION SOLUTIONS: ANOTHER LOOK AT COMPUTATIONAL ACCURACY

Flexible Techniques to Detect Typical Hidden Errors in Large Longitudinal Datasets

Low-rank matrix estimation via nonconvex optimization methods in multi-response errors-in-variables regression

Real-time Detection of Systematic Errors in Distributed Sensors

Cost-Sensitive Feature Selection of Data with Errors

Accurate Uncertainties for Deep Learning Using Calibrated Regression

Denoising and Error Correction in Wireless Sensor Networks

Detecting Data Errors with Statistical Constraints

Low-rank matrix estimation via nonconvex spectral regularized methods in errors-in-variables matrix regression

Error Detection and Uncertainty Modeling for Imprecise Data

Least absolute deviations for uncertain multivariate regression model

Error Density-dependent Empirical Risk Minimization

Low-rank matrix estimation in multi-response regression with measurement errors: Statistical and computational guarantees.