A Novel Online Hydrological Data Quality Control Approach Based on Adaptive Differential Evolution

Qun Zhao,Shicheng Cui,Yuelong Zhu,Rui Li,Xudong Zhou

DOI: https://doi.org/10.3390/math12121821

IF: 2.4

2024-06-13

Mathematics

Abstract:The quality of hydrological data has a significant impact on hydrological models, where stable and anomaly-free hydrological time series typically yield more valuable patterns. In this paper, we conduct data analysis and propose an online hydrological data quality control method based on an adaptive differential evolution algorithm according to the characteristics of hydrological data. Taking into account the characteristics of continuity, periodicity, and seasonality, we develop a Periodic Temporal Long Short-Term Memory (PT-LSTM) predictive control model. Building upon the real-time nature of the data, we apply the Adaptive Differential Evolution algorithm to optimize PT-LSTM, creating an Online Composite Predictive Control Model (OCPT-LSTM) that provides confidence intervals and recommended values for control and replacement. The experimental results demonstrate that the proposed data quality control method effectively manages data quality; detects data anomalies; provides suggested values; reduces reliance on manual intervention; provides a solid data foundation for hydrological data analysis work; and helps hydrological personnel in water resource scheduling, flood control, and other related tasks. Meanwhile, the proposed method can also be applied to the analysis of time series data in other industries.

mathematics

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve This paper aims to address several key issues in hydrological data quality control: 1. **Dependence on Manual Intervention**: - Currently, most hydrological data quality control methods are still in the theoretical research and modeling stage. Controlling data quality often requires manual intervention, lacking intelligent data quality control algorithms and models. 2. **Low Credibility of Short-term Missing Data Imputation**: - For short-term missing values caused by equipment failures, commonly used interpolation methods (such as the average method, weighted method, or simple spatial interpolation method) perform poorly, and the credibility of the imputed data is difficult to assess. 3. **Lack of Reliable Replacement Values for Anomalous Data**: - Many current methods detect anomalous data through basic checks and standard settings but fail to provide reliable replacement values for the anomalous data. To address these issues, this study proposes an intelligent hydrological data quality control method based on the continuity, periodicity, seasonality, and real-time nature of hydrological data. The aim is to reduce dependence on manual intervention, optimize hydrological data quality, and thereby improve the stability and accuracy of data mining models.

A Novel Online Hydrological Data Quality Control Approach Based on Adaptive Differential Evolution

A Robust Strategy to Account for Data Sampling Variability in the Development of Hydrological Models

Adaptive Assessment of Reservoir Scheduling to Hydrometeorological Comprehensive Dry and Wet Condition Evolution in a Multi-Reservoir Region of Southeastern China

Improved Data Splitting Methods for Data-Driven Hydrological Model Development Based on a Large Number of Catchment Samples

Self-adaptive differential evolution algorithm applied to water distribution system optimization

A Hybrid Evolutionary Data Driven Model for River Water Quality Early Warning.

Real-time Water Quality Detection Based on Fluctuation Feature Analysis with the LSTM Model

Contamination Event Detection Method Based On The Longest Common Subsequence Analysis Using Multiple Water Quality Parameters

Enhancing Water Quality Monitoring: an Integrated Machine Learning Framework with Physical Constraints for Imputation and Time Series Downscaling

Evolutionary artificial intelligence model via cooperation search algorithm and extreme learning machine for multiple scales nonstationary hydrological time series prediction

Predicting Urban Water Quality with Ubiquitous Data - A Data-driven Approach

A Hydrological Knowledge-Informed LSTM Model for Monthly Streamflow Reconstruction Using Distributed Data: Application to Typical Rivers Across the Tibetan Plateau

Water Quality Prediction Based on an Innovated Physical and Data Driving Hybrid Model at Basin Scale

A Novel Early Warning System (EWS) for Water Quality, Integrating a High-Frequency Monitoring Database with Efficient Data Quality Control Technology at a Large and Deep Lake (Lake Qiandao), China

Water Resources Quality Indicators Monitoring by Nonlinear Programming and Simulated Annealing Optimization with Ensemble Learning Approaches

An open online simulation strategy for hydrological ensemble forecasting

Data Decomposition, Seasonal Adjustment Method and Machine Learning Combined for Runoff Prediction: A Case Study

Water-Quality Assessment and Pollution-Risk Early-Warning System Based on Web Crawler Technology and LSTM

Water quality assessment of deep learning-improved comprehensive pollution index: a case study of Dagu River, Jiaozhou Bay, China

Improving the precision of monthly runoff prediction using the combined non-stationary methods in an oasis irrigation area

A novel hybrid model for long-term water quality prediction with the 'decomposition–inputs–prediction' hierarchical optimization framework