Recent gerontological developments in psychology and social sciences.

F. Carp,C. Nydegger

DOI: https://doi.org/10.1093/GERONT/15.4.368

1975-08-01

The Gerontologist

Abstract:

What problem does this paper attempt to address?

Comparison of Imputation Methods for Missing Values in Air Pollution Data: Case Study on Sydney Air Quality Index

W. M. L. K. N. Wijesekara,Liwan Liyanage

DOI: https://doi.org/10.1007/978-3-030-39442-4_20

2020-01-01

Abstract:Missing values in air quality data may lead to a substantial amount of bias and inefficiency in modeling. In this paper, we discuss six methods for dealing with missing values in univariate time series and compare their performances. The methods we discuss here are Mean Imputation, Spline Interpolation, Simple Moving Average, Exponentially Weighted Moving Average, Kalman Smoothing on Structural Time Series Models and Kalman Smoothing on Autoregressive Integrated Moving Average (ARIMA) models. The performances of these methods were compared using three different performance measures; Mean Squared Error, Coefficient of Determination and the Index of Agreement. Kalman Smoothing on Structural Time Series method is the best method among the methods considered, for imputing missing values in the context of air quality data under Missing Completely at Random (MCAR) mechanism. Kalman Smoothing on ARIMA, and Exponentially Weighted Moving Average methods also perform considerably well. Performance of Spline Interpolation decreases drastically with increased percentage of missing values. Mean Imputation performs reasonably well for smaller percentage of missing values; however, all the other methods outperform Mean Imputation regardless the number of missing values.
A transferred spatio-temporal deep model based on multi-LSTM auto-encoder for air pollution time series missing value imputation

Xiaoxia Zhang,Pengcheng Zhou

DOI: https://doi.org/10.1016/j.future.2024.03.015

IF: 7.307

2024-03-17

Future Generation Computer Systems

Abstract:Air pollution is one of the most severe problems facing the world. Research on air quality prediction and analysis of influencing factors also continues to grow. When conducting this research, valid, authentic, and high-quality air pollution data are necessary to obtain reasonable results. However, Missing values are unavoidable in multivariate time series due to multiple causes, such as sensor and communication failure. Most previous algorithms on missing data cannot effectively pay attention to air pollution's temporal and spatial mechanism, handle multiple missing patterns, or deal with high missing rates sequences. This paper proposes a new deep spatiotemporal imputation methodology to address this problem effectively, namely transferred Multiple LSTM based deep auto-encoder (TMLSTM-AE). Our idea is intuitive: train an auto-encoder to estimate the missing values. It uses spatial and time series information to fill in single missing, multiple missing, block missing, and long-interval consecutive missing in air quality data. To verify the effectiveness and priority of the proposed model, we conducted a case study in a city in Shaanxi, China. Long-interval consecutive missing and different missing patterns PM2.5 data are filled. The results indicate that the model proposed in this paper performs well and outperforms existing models for different missing patterns and long-interval consecutive missing.

computer science, theory & methods
Data analysis and preprocessing techniques for air quality prediction: a survey

Chengqing Yu,Jing Tan,Yihan Cheng,Xiwei Mi

DOI: https://doi.org/10.1007/s00477-024-02693-4

IF: 3.821

2024-03-19

Stochastic Environmental Research and Risk Assessment

Abstract:Air quality prediction technology can provide effective technical means for environmental governance. In recent years, due to the strong nonlinearity of data, there has been extensive research on data analysis and preprocessing techniques. This paper aims to comprehensively summarize and analyze the methods used in air quality forecasting, specifically focusing on four categories: data decomposition, dimensionality reduction, data correction, and spatial interpolation. Each method's purpose, characteristics, improvements, and implementation details are described in detail. The evaluation of data preprocessing methods is based on popularity, accuracy improvements, time consumption, maturity, and implementation difficulty. Among the existing methods, data decomposition and feature selection are commonly used and well-developed. However, outlier detection and spatial interpolation have limited applications and require further research. Furthermore, this paper discusses current challenges in applying these methods and future development trends, providing a valuable reference for future research.

environmental sciences,engineering, environmental,water resources, civil,statistics & probability
Median-KNN Regressor-SMOTE-Tomek Links for Handling Missing and Imbalanced Data in Air Quality Prediction

Winoto Chandra,Bambang Suprihatin,Yulia Resti

DOI: https://doi.org/10.3390/sym15040887

2023-04-10

Symmetry

Abstract:The Air Quality Index (AQI) dataset contains information on measurements of pollutants and ambient air quality conditions at certain location that can be used to predict air quality. Unfortunately, this dataset often has many missing observations and imbalanced classes. Both of these problems can affect the performance of the prediction model. In particular, predictions for the minority class are very important because inaccurate predictions can be fatal or cause big losses. Moreover, the missing data may lead to biased results. This paper proposes the single imputation of the median and the multiple imputations of the -Nearest Neighbor (KNN) regressor to handle missing values of less than or equal to 10% and more than 10%, respectively. At the same time, the SMOTE-Tomek Links address the imbalanced class. These proposed approaches to handle both issues are then used to assess the air quality prediction of the India AQI dataset using Naive Bayes (NB), KNN, and C4.5. The five treatments show that the proposed method of the Median-KNN regressor-SMOTE-Tomek Links is able to improve the performance of the India air quality prediction model. In other words, the proposed method succeeds in overcoming the problems of missing values and class imbalance.

multidisciplinary sciences
Handling missing data in near real-time environmental monitoring: A system and a review of selected methods

Yifan Zhang,Peter J. Thorburn

DOI: https://doi.org/10.1016/j.future.2021.09.033

IF: 7.307

2022-03-01

Future Generation Computer Systems

Abstract:High-frequency water quality monitoring systems provide valuable measurements for predicting the trend of water quality, warning of abnormal activities or operating hydrological models. However, missing values are prevalent due to network miscommunication, device replacement or failure. Applying datasets with missing values can lead to biased results in statistical analysis or hydrological modelling work. We develop a cloud-based data processing system combining advanced algorithms to impute monitoring data in near real-time. The system provides high compatibility for supporting different water quality variables, imputation algorithms and extensive scalability to support numerous data streams. Based on the proposed approach, we review various imputation methods which can be applied to water quality data. Overall, this work provides a systematic design of a water quality data imputation system, explores the advantages and limitations of selected data imputation methods and analyses the imputation performance of two real-time water quality monitoring systems located in both the USA and Australia. The results provide practical guidelines for data imputation applications in water quality data.
An Intelligent Missing Data Imputation Techniques: A Review

Kimseth Seu,Mi-Sun Kang,HwaMin Lee

DOI: https://doi.org/10.30630/joiv.6.1-2.935

2022-05-31

Abstract:The incomplete dataset is an unescapable problem in data preprocessing that primarily machine learning algorithms could not employ to train the model. Various data imputation approaches were proposed and challenged each other to resolve this problem. These imputations were established to predict the most appropriate value using different machine learning algorithms with various concepts. Furthermore, accurate estimation of the imputation method is exceptionally critical for some datasets to complete the missing value, especially imputing datasets in medical data. The purpose of this paper is to express the power of the distinguished state-of-the-art benchmarks, which have included the K-nearest Neighbors Imputation (KNNImputer) method, Bayesian Principal Component Analysis (BPCA) Imputation method, Multiple Imputation by Center Equation (MICE) Imputation method, Multiple Imputation with denoising autoencoder neural network (MIDAS) method. These methods have contributed to the achievable resolution to optimize and evaluate the appropriate data points for imputing the missing value. We demonstrate the experiment with all these imputation techniques based on the same four datasets which are collected from the hospital. Both Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) are utilized to measure the outcome of implementation and compare with each other to prove an extremely robust and appropriate method that overcomes missing data problems. As a result of the experiment, the KNNImputer and MICE have performed better than BPCA and MIDAS imputation, and BPCA has performed better than the MIDAS algorithm.
Missing Traffic Data: Comparison of Imputation Methods

Yuebiao Li,Zhiheng Li,Li

DOI: https://doi.org/10.1049/iet-its.2013.0052

IF: 2.7

2014-01-01

IET Intelligent Transport Systems

Abstract:Many traffic management and control applications require highly complete and accurate data of traffic flow. However, because of various reasons such as sensor failure or transmission error, it is common that some traffic flow data are lost. As a result, various methods were proposed by using a wide spectrum of techniques to estimate missing traffic data in the last two decades. Generally, these missing data imputation methods can be categorised into three kinds: prediction methods, interpolation methods and statistical learning methods. To assess their performance, these methods are compared from different aspects in this paper, including reconstruction errors, statistical behaviours and running speeds. Results show that statistical learning methods are more effective than the other two kinds of imputation methods when data of a single detector is utilised. Among various methods, the probabilistic principal component analysis (PPCA) yields best performance in all aspects. Numerical tests demonstrate that PPCA can be used to impute data online before making further analysis (e.g. make traffic prediction) and is robust to weather changes.
A Spatiotemporal Prediction Framework for Air Pollution Based on Deep RNN

FAN Junxiang,LI Qi,ZHU Yajie,HOU Junxiong,FENG Xiao

DOI: https://doi.org/10.5194/isprs-annals-iv-4-w2-15-2017

2017-01-01

Abstract:Time series data in practical applications always contain missing values due to sensor malfunction, network failure, outliers etc. In order to handle missing values in time series, as well as the lack of considering temporal properties in machine learning models, we propose a spatiotemporal prediction framework based on missing value processing algorithms and deep recurrent neural network (DRNN). By using missing tag and missing interval to represent time series patterns, we implement three different missing value fixing algorithms, which are further incorporated into deep neural network that consists of LSTM (Long Short-term Memory) layers and fully connected layers. Real-world air quality and meteorological datasets (Jingjinji area, China) are used for model training and testing. Deep feed forward neural networks (DFNN) and gradient boosting decision trees (GBDT) are trained as baseline models against the proposed DRNN. Performances of three missing value fixing algorithms, as well as different machine learning models are evaluated and analysed. Experiments show that the proposed DRNN framework outperforms both DFNN and GBDT, therefore validating the capacity of the proposed framework. Our results also provides useful insights for better understanding of different strategies that handle missing values.
Selection of statistical technique for imputation of single site-univariate and multisite–multivariate methods for particulate pollutants time series data with long gaps and high missing percentage

Priti K,Kaushlesh Singh Shakya,Prashant Kumar

DOI: https://doi.org/10.1007/s11356-023-27659-x

IF: 5.8

2023-05-24

Environmental Science and Pollution Research

Abstract:Monitoring air contaminants has become essential to exposure science, toxicology, and public health research. However, missing values are common while monitoring air contaminants, especially in resource-constrained settings such as power cuts, calibration, and sensor failure. In contaminants monitoring, evaluating existing imputation techniques for dealing with recurrent periods of missing and unobserved data are limited. The proposed study aims to perform a statistical evaluation of six univariate and four multivariate time series imputation methods. The univariate methods are based on inter-time correlation characteristics, and the multivariate approach considers muti-site to impute missing data. The present study retrieved data from 38 ground-based monitoring stations for particulate pollutants in Delhi for 4 years. For univariate methods, missing values were simulated under 0–20% (5%, 10%, 15%, and 20%), and high 40%, 60%, and 80% missing levels having long gaps. Before evaluating multivariate methods, input data underwent pre-processing steps: selecting the target station to be imputed, choosing covariates based on the spatial correlation between multiple sites, and framing a combination of target and neighbouring stations (covariates) under 20%, 40%, 60%, and 80%. Next, the particulate pollutants data of 1480 days is provided as input to four multivariate techniques. Finally, the performance of each algorithm was evaluated using error metrics. The results show that the long interval time series data and spatial correlation of multiple stations significantly improved outcomes for univariate and multivariate time series methods. The univariate Kalman_arima performs well for long-missing gaps and all missing levels (except for 60–80%), yielding low error and high R 2 and d values. In contrast, multivariate MIPCA performed better than Kalman-arima for all target stations with the highest missing percentage.

environmental sciences
Novel MIA-LSTM Deep Learning Hybrid Model with Data Preprocessing for Forecasting of PM2.5

Gaurav Narkhede,Anil Hiwale,Bharat Tidke,Chetan Khadse

DOI: https://doi.org/10.3390/a16010052

2023-01-13

Algorithms

Abstract:Day by day pollution in cities is increasing due to urbanization. One of the biggest challenges posed by the rapid migration of inhabitants into cities is increased air pollution. Sustainable Development Goal 11 indicates that 99 percent of the world's urban population breathes polluted air. In such a trend of urbanization, predicting the concentrations of pollutants in advance is very important. Predictions of pollutants would help city administrations to take timely measures for ensuring Sustainable Development Goal 11. In data engineering, imputation and the removal of outliers are very important steps prior to forecasting the concentration of air pollutants. For pollution and meteorological data, missing values and outliers are critical problems that need to be addressed. This paper proposes a novel method called multiple iterative imputation using autoencoder-based long short-term memory (MIA-LSTM) which uses iterative imputation using an extra tree regressor as an estimator for the missing values in multivariate data followed by an LSTM autoencoder for the detection and removal of outliers present in the dataset. The preprocessed data were given to a multivariate LSTM for forecasting PM2.5 concentration. This paper also presents the effect of removing outliers and missing values from the dataset as well as the effect of imputing missing values in the process of forecasting the concentrations of air pollutants. The proposed method provides better results for forecasting with a root mean square error (RMSE) value of 9.8883. The obtained results were compared with the traditional gated recurrent unit (GRU), 1D convolutional neural network (CNN), and long short-term memory (LSTM) approaches for a dataset of the Aotizhonhxin area of Beijing in China. Similar results were observed for another two locations in China and one location in India. The results obtained show that imputation and outlier/anomaly removal improve the accuracy of air pollution forecasting.
Evaluation of data imputation approaches for multi-stream building systems data1

Ojas Pradhan,Jin Wen,David Hälleberg,Zhelun Chen,Noresh Varman,Jiajing Huang,Teresa Wu,K. Selçuk Candan,Zheng O’Neill

DOI: https://doi.org/10.1080/23744731.2024.2351311

2024-05-25

Science and Technology for the Built Environment

Abstract:Increasing advancements in building digitization, smart sensing, and metering technologies have allowed large amounts of timeseries data to be collected for monitoring, analyzing, and controlling building systems. However, due to sensor or communication failures, the data collected are often incomplete and poor in quality. Data imputation approaches to replace the missing values, specifically based on either statistical or predictive models have been widely adopted for multivariate datasets in other domains. It is hence of interest to find an effective way to impute timeseries data collected from a building system. In this paper, we evaluate multiple data imputation approaches using data collected from a medium sized building situated in Stockholm, Sweden and a small commercial building from the ASHRAE RP-1312 research project. Sensors with widely varying characteristics from the case study buildings were selected to evaluate the imputation methods. The imputation accuracy and the impact of each chosen imputation method on information entropy, short-term building forecasting model performance, and fault detection strategy were evaluated. Results demonstrate that incorporating time-lagged cross correlations within a k -nearest neighbor ( k NN) model provide the most accurate imputations without affecting the quality of subsequent data analysis.

engineering, mechanical,thermodynamics,construction & building technology
Managing Large Dataset Gaps in Urban Air Quality Prediction: DCU-Insight-AQ at MediaEval 2022

Dinh Viet Cuong,Phuc H. Le-Khac,Adam Stapleton,Elke Eichlemann,Mark Roantree,Alan F. Smeaton

DOI: https://doi.org/10.48550/arXiv.2212.10273

IF: 5.414

2022-12-19

Machine Learning

Abstract:Calculating an Air Quality Index (AQI) typically uses data streams from air quality sensors deployed at fixed locations and the calculation is a real time process. If one or a number of sensors are broken or offline, then the real time AQI value cannot be computed. Estimating AQI values for some point in the future is a predictive process and uses historical AQI values to train and build models. In this work we focus on gap filling in air quality data where the task is to predict the AQI at 1, 5 and 7 days into the future. The scenario is where one or a number of air, weather and traffic sensors are offline and explores prediction accuracy under such situations. The work is part of the MediaEval'2022 Urban Air: Urban Life and Air Pollution task submitted by the DCU-Insight-AQ team and uses multimodal and crossmodal data consisting of AQI, weather and CCTV traffic images for air pollution prediction.
Recover Missing Sensor Data with Iterative Imputing Network

Jingguang Zhou,Zili Huang

DOI: https://doi.org/10.48550/arXiv.1711.07878

2017-11-20

Abstract:Sensor data has been playing an important role in machine learning tasks, complementary to the human-annotated data that is usually rather costly. However, due to systematic or accidental mis-operations, sensor data comes very often with a variety of missing values, resulting in considerable difficulties in the follow-up analysis and visualization. Previous work imputes the missing values by interpolating in the observational feature space, without consulting any latent (hidden) dynamics. In contrast, our model captures the latent complex temporal dynamics by summarizing each observation's context with a novel Iterative Imputing Network, thus significantly outperforms previous work on the benchmark Beijing air quality and meteorological dataset. Our model also yields consistent superiority over other methods in cases of different missing rates.

Machine Learning
An Experimental Survey of Missing Data Imputation Algorithms

Xiaoye Miao,Yangyang Wu,Lu Chen,Yunjun Gao,Jianwei Yin

DOI: https://doi.org/10.1109/tkde.2022.3186498

IF: 9.235

2022-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Due to the ubiquity of missing data, data imputation has received extensive attention in the past decades. It is a well-recognized problem impacting almost all fields of scientific study. Existing imputation algorithms differ in problem settings, model selection, and data evaluation. There is a lack of systematic comparison study among imputation algorithms. In this paper, we survey this interesting and evolving research topic by broadly reviewing and experimentally comparing the state-of-the-art missing data imputation algorithms. We analyze and categorize 19 imputation algorithms. Extensive experiments over 15 real-world benchmark datasets are conducted under various settings of data types, missing mechanisms, missing rates, dataset/model parameters, as well as the post-imputation prediction task. We shed light on a series of constructive insights on imputation algorithms to tackle imputation problem in real-life scenarios. Moreover, we put forward promising future directions for data imputation problem.
Comparison of Missing Data Imputation Methods in Time Series Forecasting

Hyun Ahn,Kyunghee Sun,Kwanghoon Pio Kim

DOI: https://doi.org/10.32604/cmc.2022.019369

2022-01-01

Abstract:Time series forecasting has become an important aspect of data analysis and has many real-world applications. However, undesirable missing values are often encountered, which may adversely affect many forecasting tasks. In this study, we evaluate and compare the effects of imputation methods for estimating missing values in a time series. Our approach does not include a simulation to generate pseudo-missing data, but instead perform imputation on actual missing data and measure the performance of the forecasting model created therefrom. In an experiment, therefore, several time series forecasting models are trained using different training datasets prepared using each imputation method. Subsequently, the performance of the imputation methods is evaluated by comparing the accuracy of the forecasting models. The results obtained from a total of four experimental cases show that the k-nearest neighbor technique is the most effective in reconstructing missing data and contributes positively to time series forecasting compared with other imputation methods.
A Benchmark for Data Imputation Methods

Sebastian Jäger,Arndt Allhorn,Felix Bießmann

DOI: https://doi.org/10.3389/fdata.2021.693674

2021-07-08

Abstract:With the increasing importance and complexity of data pipelines, data quality became one of the key challenges in modern software applications. The importance of data quality has been recognized beyond the field of data engineering and database management systems (DBMSs). Also, for machine learning (ML) applications, high data quality standards are crucial to ensure robust predictive performance and responsible usage of automated decision making. One of the most frequent data quality problems is missing values. Incomplete datasets can break data pipelines and can have a devastating impact on downstream ML applications when not detected. While statisticians and, more recently, ML researchers have introduced a variety of approaches to impute missing values, comprehensive benchmarks comparing classical and modern imputation approaches under fair and realistic conditions are underrepresented. Here, we aim to fill this gap. We conduct a comprehensive suite of experiments on a large number of datasets with heterogeneous data and realistic missingness conditions, comparing both novel deep learning approaches and classical ML imputation methods when either only test or train and test data are affected by missing data. Each imputation method is evaluated regarding the imputation quality and the impact imputation has on a downstream ML task. Our results provide valuable insights into the performance of a variety of imputation methods under realistic conditions. We hope that our results help researchers and engineers to guide their data preprocessing method selection for automated data quality improvement.
Multiview data fusion technique for missing value imputation in multisensory air pollution dataset

Middya, Asif Iqbal,Roy, Sarbani

DOI: https://doi.org/10.1007/s12652-024-04816-9

IF: 3.662

2024-06-05

Journal of Ambient Intelligence and Humanized Computing

Abstract:The missing readings in various sensors of air pollution monitoring stations is a common issue. Those missing sensor readings may greatly influence the performance of monitoring and analysis of air pollution data. To address this problem, in this paper, a multi-view based missing value (MV) imputation method called MVDI ( M ulti- V iew D ata I mputation) is proposed for air pollution related time series data. MVDI combines four models namely LSTM (Long-Short Term Memory), IDS (Inverse Distance Squared), SVR (Support Vector Regressor), and KNN (K-Nearest Neighbors) to estimate MVs. These four models are mainly employed to capture the variations in data from different views of the dataset. Here, different views represent different portions (subsets) of the actual dataset. The estimates of MVs from all the views are combined using a kernel function to get an overall result. The proposed model MVDI is evaluated on real-world air pollution dataset in terms of RMSE, MAE, MAPE, and R 2 . The experimental results show that MVDI dominates over the baseline methods namely AR (AutoRegressive), ARIMA (AutoRegressive Integrated Moving Average), RFR (Random Forest Regressor), ANN (Artificial Neural Network), LI (Linear Interpolation), NN (Nearest Neighbors), MI (Mean Imputation), CNN (Convolutional Neural Network), ConvLSTM (Convolutional LSTM).

computer science, information systems,telecommunications, artificial intelligence
Missing Value Imputation of Wireless Sensor Data for Environmental Monitoring

Thomas Decorte,Steven Mortier,Jonas J. Lembrechts,Filip J. R. Meysman,Steven Latré,Erik Mannens,Tim Verdonck

DOI: https://doi.org/10.3390/s24082416

IF: 3.9

2024-04-10

Sensors

Abstract:Over the past few years, the scale of sensor networks has greatly expanded. This generates extended spatiotemporal datasets, which form a crucial information resource in numerous fields, ranging from sports and healthcare to environmental science and surveillance. Unfortunately, these datasets often contain missing values due to systematic or inadvertent sensor misoperation. This incompleteness hampers the subsequent data analysis, yet addressing these missing observations forms a challenging problem. This is especially the case when both the temporal correlation of timestamps within a single sensor and the spatial correlation between sensors are important. Here, we apply and evaluate 12 imputation methods to complete the missing values in a dataset originating from large-scale environmental monitoring. As part of a large citizen science project, IoT-based microclimate sensors were deployed for six months in 4400 gardens across the region of Flanders, generating 15-min recordings of temperature and soil moisture. Methods based on spatial recovery as well as time-based imputation were evaluated, including Spline Interpolation, MissForest, MICE, MCMC, M-RNN, BRITS, and others. The performance of these imputation methods was evaluated for different proportions of missing data (ranging from 10% to 50%), as well as a realistic missing value scenario. Techniques leveraging the spatial features of the data tend to outperform the time-based methods, with matrix completion techniques providing the best performance. Our results therefore provide a tool to maximize the benefit from costly, large-scale environmental monitoring efforts.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
Data-Driven Air Quality Characterization for Urban Environments: A Case Study

Yuchao Zhou,Suparna De,Gideon Ewa,Charith Perera,Klaus Moessner

DOI: https://doi.org/10.1109/access.2018.2884647

IF: 3.9

2018-01-01

IEEE Access

Abstract:The economic and social impact of poor air quality in towns and cities is increasingly being recognized, together with the need for effective ways of creating awareness of real-time air quality levels and their impact on human health. With local authority maintained monitoring stations being geographically sparse and the resultant datasets also featuring missing labels, computational data-driven mechanisms are needed to address the data sparsity challenge. In this paper, we propose a machine learning-based method to accurately predict the air quality index, using environmental monitoring data together with meteorological measurements. To do so, we develop an air quality estimation framework that implements a neural network that is enhanced with a novel non-linear autoregressive neural network with exogenous input model, especially designed for time series prediction. The framework is applied to a case study featuring different monitoring sites in London, with comparisons against other standard machine-learning-based predictive algorithms showing the feasibility and robust performance of the proposed method for different kinds of areas within an urban region.

computer science, information systems,telecommunications,engineering, electrical & electronic
A Spatiotemporal Approach for Traffic Data Imputation with Complicated Missing Patterns

Huiping Li,Meng Li,Xi Lin,Fang He,Yinhai Wang

DOI: https://doi.org/10.1016/j.trc.2020.102730

IF: 9.022

2020-01-01

Transportation Research Part C Emerging Technologies

Abstract:With the advent of intelligent transportation systems (ITS), spatiotemporal traffic data has gained growing importance in real-time monitoring, prediction, and control of traffic. However, in practical implementations, data collection devices are often faced with malfunctions caused by various unpredictable disruptions, thereby resulting in the so-called "missing value problems." In realistic cases, the disruptions to the data collection devices are often associated with some key events (e.g., power cut and natural disasters), in addition, along with other disruptions the missing value problem could be in a complicated manner with both randomly and completely missing patterns. To perform the imputation task with such complicated missing patterns, we propose a hybrid spatiotemporal method which utilizes the time series properties by "prophet" model and captures the spatial residuals information by iterative random forest model. The spatiotemporal method first applies the temporal part to fill the missing value and then adopts the spatial part to acquire the residual component of the missing values. The results of the two components are integrated into the final imputations. Based on the PeMS freeway dataset (PeMS, 2019) and an urban road dataset under extensive artificially designed scenarios like randomly, clustered non-completely and completely missing patterns, we test our proposed approach with some existing techniques such as K-Nearest Neighbor (KNN), Seasonal-Trend decomposition using Loess (STL), Bayesian tensor decomposition, Denoising AutoEncoder (DAE). The test results indicate that the hybrid method achieves the best imputation quality for most missing patterns, particularly for those with completely or hybrid missing patterns. Furthermore, the hybrid model still performs well under extreme missing rates as high as 0.9, which validates the robustness of the model in extreme situations.

Recent gerontological developments in psychology and social sciences.

Comparison of Imputation Methods for Missing Values in Air Pollution Data: Case Study on Sydney Air Quality Index

A transferred spatio-temporal deep model based on multi-LSTM auto-encoder for air pollution time series missing value imputation

Data analysis and preprocessing techniques for air quality prediction: a survey

Median-KNN Regressor-SMOTE-Tomek Links for Handling Missing and Imbalanced Data in Air Quality Prediction

Handling missing data in near real-time environmental monitoring: A system and a review of selected methods

An Intelligent Missing Data Imputation Techniques: A Review

Missing Traffic Data: Comparison of Imputation Methods

A Spatiotemporal Prediction Framework for Air Pollution Based on Deep RNN

Selection of statistical technique for imputation of single site-univariate and multisite–multivariate methods for particulate pollutants time series data with long gaps and high missing percentage

Novel MIA-LSTM Deep Learning Hybrid Model with Data Preprocessing for Forecasting of PM2.5

Evaluation of data imputation approaches for multi-stream building systems data1

Managing Large Dataset Gaps in Urban Air Quality Prediction: DCU-Insight-AQ at MediaEval 2022

Recover Missing Sensor Data with Iterative Imputing Network

An Experimental Survey of Missing Data Imputation Algorithms

Comparison of Missing Data Imputation Methods in Time Series Forecasting

A Benchmark for Data Imputation Methods

Multiview data fusion technique for missing value imputation in multisensory air pollution dataset

Missing Value Imputation of Wireless Sensor Data for Environmental Monitoring

Data-Driven Air Quality Characterization for Urban Environments: A Case Study

A Spatiotemporal Approach for Traffic Data Imputation with Complicated Missing Patterns