Probabilistic Automatic Outlier Detection for Surface Air Quality Measurements from the China National Environmental Monitoring Network
Huangjian Wu,Xiao Tang,Zifa Wang,Lin Wu,Miaomiao Lu,Lianfang Wei,Jiang Zhu
DOI: https://doi.org/10.1007/s00376-018-8067-9
IF: 5.8
2018-10-27
Advances in Atmospheric Sciences
Abstract:Although quality assurance and quality control procedures are routinely applied in most air quality networks, outliers can still occur due to instrument malfunctions, the influence of harsh environments and the limitation of measuring methods. Such outliers pose challenges for data-powered applications such as data assimilation, statistical analysis of pollution characteristics and ensemble forecasting. Here, a fully automatic outlier detection method was developed based on the probability of residuals, which are the discrepancies between the observed and the estimated concentration values. The estimation can be conducted using filtering—or regressions when appropriate—to discriminate four types of outliers characterized by temporal and spatial inconsistency, instrument-induced low variances, periodic calibration exceptions, and less PM10 than PM2.5 in concentration observations, respectively. This probabilistic method was applied to detect all four types of outliers in hourly surface measurements of six pollutants (PM2.5, PM10, SO2, NO2, CO and O3) from 1436 stations of the China National Environmental Monitoring Network during 2014–16. Among the measurements, 0.65%–5.68% are marked as outliers, with PM10 and CO more prone to outliers. Our method successfully identifies a trend of decreasing outliers from 2014 to 2016, which corresponds to known improvements in the quality assurance and quality control procedures of the China National Environmental Monitoring Network. The outliers can have a significant impact on the annual mean concentrations of PM2.5, with differences exceeding 10 μg m−3 at 66 sites.
meteorology & atmospheric sciences