Efficient missing data imputing for traffic flow by considering temporal and spatial dependence

Li Li,Yuebiao Li,Zhiheng Li

DOI: https://doi.org/10.1016/j.trc.2013.05.008

2013-01-01

Abstract:•We enhance PPCA method by considering multiple sensors’ measurements.•We show imputing errors can be reduced by using temporal–spatial dependence.•We suggest a simple yet effective way to embed temporal–spatial dependence.

What problem does this paper attempt to address?

Improving the Traffic Data Imputation Accuracy Using Temporal and Spatial Information

Ningyu Zhao,Zhiheng Li,Yuebiao Li

DOI: https://doi.org/10.1109/icicta.2014.83

2014-01-01

Abstract:The missing data problem exists in a lot of transportation systems while many traffic applications require accurate traffic data to work well. So, various methods were proposed for traffic data imputation in recent years. However, most of these approaches only consider the information of a single detector. In this paper, we use the temporal and spatial information collected from multiple detectors to improve the traffic data imputation accuracy. Specially, we consider Probabilistic Principal Component Analysis (PPCA) and Mixed Probabilistic Principal Component Analysis (MPPCA), which had been proven to be useful methods to handle data imputation. We examine the performance of these two methods using the temporal and spatial information. Tests show that when only using the spatial information, the errors of these two methods are reduced and MPPCA works even better than PPCA. Moreover, when both of the temporal and spatial information from multiple detectors are used, more imputation accuracy is obtained than only using the spatial information for MPPCA.
Comparison on PPCA, KPPCA and MPPCA based missing data imputing for traffic flow

Li Yuebiao,Li Zhiheng,Li Li,Zhang Yi,Jin Maojing

DOI: https://doi.org/10.1061/9780784413036.155

2013-01-01

Abstract:In recent studies, the Probabilistic Principal Component Analysis (PPCA) for imputing missing data was shown to be a good tool for traffic flow data processing. The PPCA method has two major benefits: it results in significantly smaller reconstruction errors and much less computation time, which make it outperform the conventional historical and regression imputing methods. In this paper, the possibility of applying more complex PPCA methods, e.g. Kernel Probabilistic Principal Component Analysis (KPPCA) or Mixed Probabilistic Principal Component Analysis (MPPCA), to impute missing data is explored. Considering reconstruction errors and computation time cost, test results show that the basic PPCA method is still our first choice in missing data imputing for traffic flow for online systems. © 2013 American Society of Civil Engineers.
PPCA-based missing data imputation for traffic flow volume

QuLi,LiLi,ZhangYi,HuJianming

IF: 8.5

2009-01-01

IEEE Transactions on Intelligent Transportation Systems

Abstract:The missing data problem greatly affects traffic analysis. In this paper, we put forward a new reliable method called probabilistic principal component analysis (PPCA) to impute the missing flow vo...
PPCA-Based Missing Data Imputation for Traffic Flow Volume: A Systematical Approach

Li Qu,Li,Yi Zhang,Jianming Hu

DOI: https://doi.org/10.1109/tits.2009.2026312

IF: 8.5

2009-01-01

IEEE Transactions on Intelligent Transportation Systems

Abstract:The missing data problem greatly affects traffic analysis. In this paper, we put forward a new reliable method called probabilistic principal component analysis (PPCA) to impute the missing flow volume data based on historical data mining. First, we review the current missing data-imputation method and why it may fail to yield acceptable results in many traffic flow applications. Second, we examine the statistical properties of traffic flow volume time series. We show that the fluctuations of traffic flow are Gaussian type and that principal component analysis (PCA) can be used to retrieve the features of traffic flow. Third, we discuss how to use a robust PCA to filter out the abnormal traffic flow data that disturb the imputation process. Finally, we recall the theories of PPCA/Bayesian PCA-based imputation algorithms and compare their performance with some conventional methods, including the nearest/mean historical imputation methods and the local interpolation/regression methods. The experiments prove that the PPCA method provides significantly better performance than the conventional methods, reducing the root-mean-square imputation error by at least 25%.
Missing Traffic Data: Comparison of Imputation Methods

Yuebiao Li,Zhiheng Li,Li

DOI: https://doi.org/10.1049/iet-its.2013.0052

IF: 2.7

2014-01-01

IET Intelligent Transport Systems

Abstract:Many traffic management and control applications require highly complete and accurate data of traffic flow. However, because of various reasons such as sensor failure or transmission error, it is common that some traffic flow data are lost. As a result, various methods were proposed by using a wide spectrum of techniques to estimate missing traffic data in the last two decades. Generally, these missing data imputation methods can be categorised into three kinds: prediction methods, interpolation methods and statistical learning methods. To assess their performance, these methods are compared from different aspects in this paper, including reconstruction errors, statistical behaviours and running speeds. Results show that statistical learning methods are more effective than the other two kinds of imputation methods when data of a single detector is utilised. Among various methods, the probabilistic principal component analysis (PPCA) yields best performance in all aspects. Numerical tests demonstrate that PPCA can be used to impute data online before making further analysis (e.g. make traffic prediction) and is robust to weather changes.
A Spatiotemporal Approach for Traffic Data Imputation with Complicated Missing Patterns

Huiping Li,Meng Li,Xi Lin,Fang He,Yinhai Wang

DOI: https://doi.org/10.1016/j.trc.2020.102730

IF: 9.022

2020-01-01

Transportation Research Part C Emerging Technologies

Abstract:With the advent of intelligent transportation systems (ITS), spatiotemporal traffic data has gained growing importance in real-time monitoring, prediction, and control of traffic. However, in practical implementations, data collection devices are often faced with malfunctions caused by various unpredictable disruptions, thereby resulting in the so-called "missing value problems." In realistic cases, the disruptions to the data collection devices are often associated with some key events (e.g., power cut and natural disasters), in addition, along with other disruptions the missing value problem could be in a complicated manner with both randomly and completely missing patterns. To perform the imputation task with such complicated missing patterns, we propose a hybrid spatiotemporal method which utilizes the time series properties by "prophet" model and captures the spatial residuals information by iterative random forest model. The spatiotemporal method first applies the temporal part to fill the missing value and then adopts the spatial part to acquire the residual component of the missing values. The results of the two components are integrated into the final imputations. Based on the PeMS freeway dataset (PeMS, 2019) and an urban road dataset under extensive artificially designed scenarios like randomly, clustered non-completely and completely missing patterns, we test our proposed approach with some existing techniques such as K-Nearest Neighbor (KNN), Seasonal-Trend decomposition using Loess (STL), Bayesian tensor decomposition, Denoising AutoEncoder (DAE). The test results indicate that the hybrid method achieves the best imputation quality for most missing patterns, particularly for those with completely or hybrid missing patterns. Furthermore, the hybrid model still performs well under extreme missing rates as high as 0.9, which validates the robustness of the model in extreme situations.
A Hybrid Data-Driven Framework for Spatiotemporal Traffic Flow Data Imputation

Peixiao Wang,Tao Hu,Fei Gao,Ruijie Wu,Wei Guo,Xinyan Zhu

DOI: https://doi.org/10.1109/jiot.2022.3151238

IF: 10.6

2022-01-01

IEEE Internet of Things Journal

Abstract:An accurate estimation of missing data in traffic flow is crucial in urban planning, intelligent transportation, economic geography, and other fields. Thus, improving the data quality of traffic flow is a necessary step in data modeling. Most existing studies use data-driven models to determine spatiotemporal patterns in traffic flow data and fill in the missing information automatically. However, simple data-driven models have unsatisfactory results for describing complex patterns in traffic flow and filling in missing data. This study treated the complex patterns in traffic flow as integrating multiple simple patterns and proposed a hybrid missing data imputation framework called ST-PTD. We used a specific time-series analysis to mine periodic patterns and proposed a novel matrix decomposition method to describe the trend of the traffic flow data. Furthermore, we fused the periodic and trend characteristics of the missing data using a novel dendritic neural network. We applied the framework in actual traffic flow data sets collected in Wuhan, China. The results showed that the ST-PTD framework outperformed the eight existing baselines in terms of imputation accuracy.

computer science, information systems,telecommunications,engineering, electrical & electronic
A Bipartite Graph Based Method for Traffic Continuous Data Imputation

Peng Jiaheng,Guan Tong,Liang Jun

DOI: https://doi.org/10.1109/cac59555.2023.10451480

2023-01-01

Abstract:Traffic information missing conditions are common in Intelligent Transportation System and would cause huge challenges for sequential traffic tasks, such as traffic speed prediction. Rare models are designed to tackle the long-range continuous data missing problems. Thus, in this paper, we propose a Bipartite Graph-based message passing Imputation Network (BGIN) to reconstruct traffic features data of nodes. In BGIN, the nodes are divided into two types: known nodes that contain traffic information and imputed nodes without data. First, a known information depth extraction module is given to achieve the messages passing among known nodes. Then, we construct a bipartite graph to represent the correlations between known and imputed nodes. Bipartite graph propagation module propagates the known information to imputed nodes. Finally, a imputation output module aggregates the messages to complete the imputation task. To evaluate the performance of our method, extensive experiments are conducted on a newly traffic imputation dataset called PeMS-impute. Experiment results demonstrate that BGIN achieves the best imputation effects than some graph based traffic prediction models and imputation models.
A Bpca Based Missing Value Imputing Method For Traffic Flow Volume Data

Li Qu,Yi Zhang,Jianming Hu,Liyan Jia,Li Li.

DOI: https://doi.org/10.1109/IVS.2008.4621153

2008-01-01

Abstract:Missing data problem widely exists in many traffic information systems, which brings great trouble to further studies. To solve this problem, this paper proposes a Bayesian Principal Component Analysis (BPCA) based missing value imputing method to impute the incomplete traffic flow volume data collected in Beijing. Intuitively, this method takes an appropriate tradeoff between the historical and periodic information when imputing missing data. Experiments prove that the proposed method provides significant better imputing performance than two other frequently used imputing methods: historical imputing and spline imputing.
Adaptive Rolling Smoothing with Heterogeneous Data for Traffic State Estimation and Prediction.

Xiqun Chen,Shuaichao Zhang,Li,Liang Li

DOI: https://doi.org/10.1109/tits.2018.2847024

IF: 8.5

2018-01-01

IEEE Transactions on Intelligent Transportation Systems

Abstract:Spatial-temporal traffic state estimation and the prediction of urban expressways is a vital component of traffic management and information systems. The adaptive smoothing method is one of the most frequently used approaches to estimate traffic states. However, the fixed filter parameters used in existing approaches sometimes fail to characterize traffic dynamics well. To better capture generation, propagation, and mitigation dynamics of traffic congestion, we propose an adaptive rolling smoothing (ARS) approach by dynamically tuning the filter parameters in a rolling horizon scheme for online applications. The fusion of heterogeneous traffic data combines aggregate traffic measurements (e.g., traffic flow rate, time occupancy, and speed collected by remote microwave sensors) and disaggregate information (e.g., timestamps of individual vehicles detected by license plate recognition cameras). A nonlinear traffic flow filter based on the virtual trajectory algorithm is established to reconstruct the spatial-temporal traffic state and estimate experienced travel times of individual vehicles. The results demonstrate the capability and effectiveness of the proposed ARS approach in the historical traffic state estimation and short-term traffic flow prediction. Complicated traffic states of weaving, merging, and diverging segments can he well distinguished by reconstructing time-space speed diagrams. The proposed approach can be extended to develop efficient missing data imputation algorithms and hierarchical control strategies for heterogeneously congested urban expressways.
A Kriging based spatiotemporal approach for traffic volume data imputation

Hongtai Yang,Jianjiang Yang,Lee D Han,Xiaohan Liu,Li Pu,Shih-Miao Chin,Ho-Ling Hwang

DOI: https://doi.org/10.1371/journal.pone.0195957

IF: 3.7

2018-04-17

PLoS ONE

Abstract:Along with the rapid development of Intelligent Transportation Systems, traffic data collection technologies have progressed fast. The emergence of innovative data collection technologies such as remote traffic microwave sensor, Bluetooth sensor, GPS-based floating car method, and automated license plate recognition, has significantly increased the variety and volume of traffic data. Despite the development of these technologies, the missing data issue is still a problem that poses great challenge for data based applications such as traffic forecasting, real-time incident detection, dynamic route guidance, and massive evacuation optimization. A thorough literature review suggests most current imputation models either focus on the temporal nature of the traffic data and fail to consider the spatial information of neighboring locations or assume the data follow a certain distribution. These two issues reduce the imputation accuracy and limit the use of the corresponding imputation methods respectively. As a result, this paper presents a Kriging based data imputation approach that is able to fully utilize the spatiotemporal correlation in the traffic data and that does not assume the data follow any distribution. A set of scenarios with different missing rates are used to evaluate the performance of the proposed method. The performance of the proposed method was compared with that of two other widely used methods, historical average and K-nearest neighborhood. Comparison results indicate that the proposed method has the highest imputation accuracy and is more flexible compared to other methods.
A Deep Learning Framework for Traffic Data Imputation Considering Spatiotemporal Dependencies

Li Jiang,Ting Zhang,Qiruyi Zuo,Chenyu Tian,George P. Chan,Wai Kin,Chan

DOI: https://doi.org/10.48550/arXiv.2304.09182

2023-04-18

Abstract:Spatiotemporal (ST) data collected by sensors can be represented as multi-variate time series, which is a sequence of data points listed in an order of time. Despite the vast amount of useful information, the ST data usually suffer from the issue of missing or incomplete data, which also limits its applications. Imputation is one viable solution and is often used to prepossess the data for further applications. However, in practice, n practice, spatiotemporal data imputation is quite difficult due to the complexity of spatiotemporal dependencies with dynamic changes in the traffic network and is a crucial prepossessing task for further applications. Existing approaches mostly only capture the temporal dependencies in time series or static spatial dependencies. They fail to directly model the spatiotemporal dependencies, and the representation ability of the models is relatively limited.

Machine Learning,Artificial Intelligence
A Latent-Factor-Model-Based Approach for Traffic Data Imputation with Road Network Information

Xing Su,Wenjie Sun,Chenting Song,Zhi Cai,Limin Guo

DOI: https://doi.org/10.3390/ijgi12090378

IF: 3.4

2023-09-16

ISPRS International Journal of Geo-Information

Abstract:With the rapid development of the economy, car ownership has grown rapidly, which causes many traffic problems. In recent years, intelligent transportation systems have been used to solve various traffic problems. To achieve effective and efficient traffic management, intelligent transportation systems need a large amount of complete traffic data. However, the current traffic data collection methods result in different forms of missing data. In the last twenty years, although many approaches have been proposed to impute missing data based on different mechanisms, these all have their limitations, which leads to low imputation accuracy, especially when the collected traffic data have a large amount of missing values. To this end, this paper proposes a latent-factor-model-based approach to impute the missing traffic data. In the proposed approach, the spatial information of the road network is first combined with the spatiotemporal matrix of the original traffic data. Then, the latent-factor-model-based algorithm is employed to impute the missing data in the combined matrix of the traffic data. Based on the real traffic data from METR-LA, we found that the imputation accuracy of the proposed approach was better than that of most of the current traffic-data-imputation approaches, especially when the original traffic data are limited.

geography, physical,remote sensing,computer science, information systems
An Experimental Evaluation of Imputation Models for Spatial-Temporal Traffic Data

Shengnan Guo,Tonglong Wei,Yiheng Huang,Miaomiao Zhao,Ran Chen,Yan Lin,Youfang Lin,Huaiyu Wan

2024-12-06

Abstract:Traffic data imputation is a critical preprocessing step in intelligent transportation systems, enabling advanced transportation services. Despite significant advancements in this field, selecting the most suitable model for practical applications remains challenging due to three key issues: 1) incomprehensive consideration of missing patterns that describe how data loss along spatial and temporal dimensions, 2) the lack of test on standardized datasets, and 3) insufficient evaluations. To this end, we first propose practice-oriented taxonomies for missing patterns and imputation models, systematically identifying all possible forms of real-world traffic data loss and analyzing the characteristics of existing models. Furthermore, we introduce a unified benchmarking pipeline to comprehensively evaluate 10 representative models across various missing patterns and rates. This work aims to provide a holistic understanding of traffic data imputation research and serve as a practical guideline.

Machine Learning
A BPCA Based Missing Value Imputation and Its Impact on Traffic Incident Prediction

Huiping Li,Yinhai Wang,Meng Li

DOI: https://doi.org/10.1061/9780784481523.177

2018-01-01

Abstract:To improve the level of road safety, many traffic incident prediction models utilizing machine learning methods are proposed. However, traffic data applied to make a prediction is often not complete, and few studies are devoted to its potential impact on the prediction accuracy. In this study, several state-of-the-art machine learning methods like extreme gradient boosting, random forest, and support vector machine are adopted to make traffic incident predictions. 123 traffic incidents, and 5 months of microwave data on an urban expressway are collected. The missing pattern in our data is discussed and imputed by 3 methods: mean interpolation, probabilistic principal component analysis (PPCA), and Bayesian principal component analysis (BPCA). A sensitivity analysis is carried out under different missing rates. The numerical test revealed that BPCA performs slightly better than PPCA, but both produce higher and more stable prediction accuracy compared with mean interpolation, especially when ensemble learning techniques are adopted.
Missing data imputation for traffic flow based on improved local least squares

Chang, Gang,Zhang, Yi,Yao, Danya

DOI: https://doi.org/10.1109/TST.2012.6216760

2012-01-01

Abstract:Complete and reliable field traffic data is vital for the planning, design, and operation of urban traffic management systems. However, traffic data is often very incomplete in many traffic information systems, which hinders effective use of the data. Methods are needed for imputing missing traffic data to minimize the effect of incomplete data on the utilization. This paper presents an improved Local Least Squares (LLS) approach to impute the incomplete data. The LLS is an improved version of the K Nearest Neighbor (KNN) method. First, the missing traffic data is replaced by a row average of the known values. Then, the vector angle and Euclidean distance are used to select the nearest neighbors. Finally, a regression step is used to get weights of the nearest neighbors and the imputation results. Traffic flow volume collected in Beijing was analyzed to compare this approach with the Bayesian Principle Component Analysis (BPCA) imputation approach. Tests show that this approach provides slightly better performance than BPCA imputation to impute missing traffic data.
A hybrid model for missing traffic flow data imputation based on clustering and attention mechanism optimizing LSTM and AdaBoost

Qiang Shang,Yingping Tang,Longjiao Yin

DOI: https://doi.org/10.1038/s41598-024-77748-1

IF: 4.6

2024-11-03

Scientific Reports

Abstract:Reliable traffic flow data is not only crucial for traffic management and planning, but also the foundation for many intelligent applications. However, the phenomenon of missing traffic flow data often occurs, so we propose an imputation model for missing traffic flow data to overcome the randomness and instability bands of traffic flow. First, k-means clustering is used to classify road segments with traffic flow belonging to the same pattern into a group to utilize the spatial characteristics of roads fully. Then, the LSTM networks optimized with an attention mechanism are used as the base learner to extract the temporal dependence of the traffic flow. Finally, the AdaBoost algorithm is used to integrate all the LSTM-attention networks into a reinforced learner to impute the missing data. To validate the effectiveness of the proposed model, we use the PeMS dataset for validation, we impute the data with missing data rate from 10 to 60% under three missing modes, and we use multiple baseline models for comparison, which confirms that our proposed model improves the stability and accuracy of imputing the missing data of the traffic flow with different scenarios.

multidisciplinary sciences
Spatial-Temporal Congestion Identification Based on Time Series Similarity Considering Missing Data

Hongsheng Qi,Meiqi Liu,Dianhai Wang,Mengwei Chen

DOI: https://doi.org/10.1371/journal.pone.0162043

IF: 3.7

2016-01-01

PLoS ONE

Abstract:Traffic congestion varies spatially and temporally. The observation of the formation, propagation and dispersion of network traffic congestion can lead to insights about the network performance, the bottleneck dynamics etc. While many researchers use the traffic flow data to reconstruct the congestion profile, the data missing problem is bypassed. Current methods either omit the missing data or supplement the missing part by average etc. Great error may be introduced during these processes. Rather than simply discarding the missing data, this research regards the data missing event as a result of either the severe congestion which prevent the floating vehicle from entering the congested area, or a type of feature of the resulting traffic flow time series. Hence a new traffic flow operational index time series similarity measurement is expected to be established as a basis of identifying the dynamic network bottleneck. The method first measures the traffic flow operational similarity between pairs of neighboring links, and then the similarity results are used to cluster the spatial-temporal congestion. In order to get the similarity under missing data condition, the measurement is implemented in a two-stage manner: firstly the so called first order similarity is calculated given that the traffic flow variables are bounded both upside and downside; then the first order similarity is aggregated to generate the second order similarity as the output. We implement the method on part of the real-world road network; the results generated are not only consistent with empirical observation, but also provide useful insights.
The Combined Short-Term Forecasting Approach to Traffic Flow Based on Non-Parametric Regression

张晓利,贺国光,陆化普

DOI: https://doi.org/10.1061/40932(246)22

2009-01-01

Abstract:A combined method based on of Principal Component Analysis (PCA), Subtraction Fuzzy Clustering (SFC), and Non-Parametric Regression (NPR) is proposed in view of nonlinearity and uncertainty for short-term forecasting of real-time traffic flow. A highly efficient case database is created from the original traffic volumes after the operations of PCA and SFC. The data-driven method of K-nearest neighbours NPR is used to make the forecasting. An emulation experiment is designed to test the validity of the method. The example results show it is better than normal NPR and meets real-time requirement.
Bidirectional Spatial-Temporal Traffic Data Imputation via Graph Attention Recurrent Neural Network

Guojiang Shen,Wenfeng Zhou,Wenyi Zhang,Nali Liu,Zhi Liu,Xiangjie Kong

DOI: https://doi.org/10.1016/j.neucom.2023.02.017

IF: 6

2023-02-18

Neurocomputing

Abstract:Spatiotemporal traffic data is increasingly important in transportation services with the development of intelligent transportation system (ITS). However, due to various unpredictable disruptions in the data collection and storage process, traffic data is often incomplete which will seriously hinder downstream tasks if not handled properly. Most existing methods for traffic data imputation either impose too strong assumptions on the data distribution or almost ignore the interdependencies across time steps and the information expressed by missingness. In this article, we propose a graph attention recurrent neural network (GARNN) for traffic data imputation. In our model, we impute data from both temporal and spatial perspectives. First, we model the observations and missingness separately via two LSTMs to treat the missingness of data as another special information distinct from observations. Then, a decay mechanism and graph attention network (GAT) are applied to learn the interdependencies across time steps and capture the spatial correlations respectively to generate temporal estimation and spatial estimation. Finally, those two estimations are integrated into the ultimate imputation. The whole process is in a bidirection. The proposed method is evaluated on two public datasets under three different missing scenarios. Experimental results show the effectiveness of the proposed model compared with other baselines.

computer science, artificial intelligence

Efficient missing data imputing for traffic flow by considering temporal and spatial dependence

Improving the Traffic Data Imputation Accuracy Using Temporal and Spatial Information

Comparison on PPCA, KPPCA and MPPCA based missing data imputing for traffic flow

PPCA-based missing data imputation for traffic flow volume

PPCA-Based Missing Data Imputation for Traffic Flow Volume: A Systematical Approach

Missing Traffic Data: Comparison of Imputation Methods

A Spatiotemporal Approach for Traffic Data Imputation with Complicated Missing Patterns

A Hybrid Data-Driven Framework for Spatiotemporal Traffic Flow Data Imputation

A Bipartite Graph Based Method for Traffic Continuous Data Imputation

A Bpca Based Missing Value Imputing Method For Traffic Flow Volume Data

Adaptive Rolling Smoothing with Heterogeneous Data for Traffic State Estimation and Prediction.

A Kriging based spatiotemporal approach for traffic volume data imputation

A Deep Learning Framework for Traffic Data Imputation Considering Spatiotemporal Dependencies

A Latent-Factor-Model-Based Approach for Traffic Data Imputation with Road Network Information

An Experimental Evaluation of Imputation Models for Spatial-Temporal Traffic Data

A BPCA Based Missing Value Imputation and Its Impact on Traffic Incident Prediction

Missing data imputation for traffic flow based on improved local least squares

A hybrid model for missing traffic flow data imputation based on clustering and attention mechanism optimizing LSTM and AdaBoost

Spatial-Temporal Congestion Identification Based on Time Series Similarity Considering Missing Data

The Combined Short-Term Forecasting Approach to Traffic Flow Based on Non-Parametric Regression

Bidirectional Spatial-Temporal Traffic Data Imputation via Graph Attention Recurrent Neural Network