PPCA-based missing data imputation for traffic flow volume

QuLi,LiLi,ZhangYi,HuJianming

IF: 8.5

2009-01-01

IEEE Transactions on Intelligent Transportation Systems

Abstract:The missing data problem greatly affects traffic analysis. In this paper, we put forward a new reliable method called probabilistic principal component analysis (PPCA) to impute the missing flow vo...

What problem does this paper attempt to address?

PPCA-Based Missing Data Imputation for Traffic Flow Volume: A Systematical Approach

Li Qu,Li,Yi Zhang,Jianming Hu

DOI: https://doi.org/10.1109/tits.2009.2026312

IF: 8.5

2009-01-01

IEEE Transactions on Intelligent Transportation Systems

Abstract:The missing data problem greatly affects traffic analysis. In this paper, we put forward a new reliable method called probabilistic principal component analysis (PPCA) to impute the missing flow volume data based on historical data mining. First, we review the current missing data-imputation method and why it may fail to yield acceptable results in many traffic flow applications. Second, we examine the statistical properties of traffic flow volume time series. We show that the fluctuations of traffic flow are Gaussian type and that principal component analysis (PCA) can be used to retrieve the features of traffic flow. Third, we discuss how to use a robust PCA to filter out the abnormal traffic flow data that disturb the imputation process. Finally, we recall the theories of PPCA/Bayesian PCA-based imputation algorithms and compare their performance with some conventional methods, including the nearest/mean historical imputation methods and the local interpolation/regression methods. The experiments prove that the PPCA method provides significantly better performance than the conventional methods, reducing the root-mean-square imputation error by at least 25%.
Comparison on PPCA, KPPCA and MPPCA based missing data imputing for traffic flow

Li Yuebiao,Li Zhiheng,Li Li,Zhang Yi,Jin Maojing

DOI: https://doi.org/10.1061/9780784413036.155

2013-01-01

Abstract:In recent studies, the Probabilistic Principal Component Analysis (PPCA) for imputing missing data was shown to be a good tool for traffic flow data processing. The PPCA method has two major benefits: it results in significantly smaller reconstruction errors and much less computation time, which make it outperform the conventional historical and regression imputing methods. In this paper, the possibility of applying more complex PPCA methods, e.g. Kernel Probabilistic Principal Component Analysis (KPPCA) or Mixed Probabilistic Principal Component Analysis (MPPCA), to impute missing data is explored. Considering reconstruction errors and computation time cost, test results show that the basic PPCA method is still our first choice in missing data imputing for traffic flow for online systems. © 2013 American Society of Civil Engineers.
A Bpca Based Missing Value Imputing Method For Traffic Flow Volume Data

Li Qu,Yi Zhang,Jianming Hu,Liyan Jia,Li Li.

DOI: https://doi.org/10.1109/IVS.2008.4621153

2008-01-01

Abstract:Missing data problem widely exists in many traffic information systems, which brings great trouble to further studies. To solve this problem, this paper proposes a Bayesian Principal Component Analysis (BPCA) based missing value imputing method to impute the incomplete traffic flow volume data collected in Beijing. Intuitively, this method takes an appropriate tradeoff between the historical and periodic information when imputing missing data. Experiments prove that the proposed method provides significant better imputing performance than two other frequently used imputing methods: historical imputing and spline imputing.
Missing Traffic Data: Comparison of Imputation Methods

Yuebiao Li,Zhiheng Li,Li

DOI: https://doi.org/10.1049/iet-its.2013.0052

IF: 2.7

2014-01-01

IET Intelligent Transport Systems

Abstract:Many traffic management and control applications require highly complete and accurate data of traffic flow. However, because of various reasons such as sensor failure or transmission error, it is common that some traffic flow data are lost. As a result, various methods were proposed by using a wide spectrum of techniques to estimate missing traffic data in the last two decades. Generally, these missing data imputation methods can be categorised into three kinds: prediction methods, interpolation methods and statistical learning methods. To assess their performance, these methods are compared from different aspects in this paper, including reconstruction errors, statistical behaviours and running speeds. Results show that statistical learning methods are more effective than the other two kinds of imputation methods when data of a single detector is utilised. Among various methods, the probabilistic principal component analysis (PPCA) yields best performance in all aspects. Numerical tests demonstrate that PPCA can be used to impute data online before making further analysis (e.g. make traffic prediction) and is robust to weather changes.
Efficient missing data imputing for traffic flow by considering temporal and spatial dependence

Li Li,Yuebiao Li,Zhiheng Li

DOI: https://doi.org/10.1016/j.trc.2013.05.008

2013-01-01

Abstract:•We enhance PPCA method by considering multiple sensors’ measurements.•We show imputing errors can be reduced by using temporal–spatial dependence.•We suggest a simple yet effective way to embed temporal–spatial dependence.
Improving the Traffic Data Imputation Accuracy Using Temporal and Spatial Information

Ningyu Zhao,Zhiheng Li,Yuebiao Li

DOI: https://doi.org/10.1109/icicta.2014.83

2014-01-01

Abstract:The missing data problem exists in a lot of transportation systems while many traffic applications require accurate traffic data to work well. So, various methods were proposed for traffic data imputation in recent years. However, most of these approaches only consider the information of a single detector. In this paper, we use the temporal and spatial information collected from multiple detectors to improve the traffic data imputation accuracy. Specially, we consider Probabilistic Principal Component Analysis (PPCA) and Mixed Probabilistic Principal Component Analysis (MPPCA), which had been proven to be useful methods to handle data imputation. We examine the performance of these two methods using the temporal and spatial information. Tests show that when only using the spatial information, the errors of these two methods are reduced and MPPCA works even better than PPCA. Moreover, when both of the temporal and spatial information from multiple detectors are used, more imputation accuracy is obtained than only using the spatial information for MPPCA.
Missing data imputation for traffic flow based on improved local least squares

Chang, Gang,Zhang, Yi,Yao, Danya

DOI: https://doi.org/10.1109/TST.2012.6216760

2012-01-01

Abstract:Complete and reliable field traffic data is vital for the planning, design, and operation of urban traffic management systems. However, traffic data is often very incomplete in many traffic information systems, which hinders effective use of the data. Methods are needed for imputing missing traffic data to minimize the effect of incomplete data on the utilization. This paper presents an improved Local Least Squares (LLS) approach to impute the incomplete data. The LLS is an improved version of the K Nearest Neighbor (KNN) method. First, the missing traffic data is replaced by a row average of the known values. Then, the vector angle and Euclidean distance are used to select the nearest neighbors. Finally, a regression step is used to get weights of the nearest neighbors and the imputation results. Traffic flow volume collected in Beijing was analyzed to compare this approach with the Bayesian Principle Component Analysis (BPCA) imputation approach. Tests show that this approach provides slightly better performance than BPCA imputation to impute missing traffic data.
Estimating Traffic Flow in Large Road Networks Based on Multi-Source Traffic Data

Pu Wang,Jiyu Lai,Zhiren Huang,Qian Tan,Tao Lin

DOI: https://doi.org/10.1109/tits.2020.2988801

IF: 8.5

2020-01-01

IEEE Transactions on Intelligent Transportation Systems

Abstract:Traffic flow data collected by traffic sensing devices is crucially important for transportation planning and transportation management. However, traffic sensing devices are typically distributed sparsely in road networks owing to their high installation and maintenance costs. The present study combines license plate recognition (LPR) data with taxi GPS trajectory data to develop a data-driven approach for estimating traffic flow in large road networks. The approach is applied to estimate traffic flow for an actual road network comprising 5,495 road segments using the traffic flow records of only 68 road segments (1.2% of the total). Five-fold cross validation is employed to verify the estimated traffic flow, and the data requirements for implementing the proposed method are analyzed. The developed data-driven approach provides an alternative and cost-efficient way of acquiring additional traffic flow information rather than installing more traffic sensing devices on roads.
A probabilistic model for missing traffic volume reconstruction based on data fusion

Xintao Yan,Yan Zhao,Henry X. Liu

DOI: https://doi.org/10.48550/arXiv.2105.02777

2021-05-07

Abstract:Traffic volume information is critical for intelligent transportation systems. It serves as a key input to transportation planning, roadway design, and traffic signal control. However, the traffic volume data collected by fixed-location sensors, such as loop detectors, often suffer from the missing data problem and low coverage problem. The missing data problem could be caused by hardware malfunction. The low coverage problem is due to the limited coverage of fixed-location sensors in the transportation network, which restrains our understanding of the traffic at the network level. To tackle these problems, we propose a probabilistic model for traffic volume reconstruction by fusing fixed-location sensor data and probe vehicle data. We apply the probabilistic principal component analysis (PPCA) to capture the correlations in traffic volume data. An innovative contribution of this work is that we also integrate probe vehicle data into the framework, which allows the model to solve both of the above-mentioned two problems. Using a real-world traffic volume dataset, we show that the proposed method outperforms state-of-the-art methods for the extensively studied missing data problem. Moreover, for the low coverage problem, which cannot be handled by most existing methods, the proposed model can also achieve high accuracy. The experiments also show that even when the missing ratio reaches 80%, the proposed method can still give an accurate estimate of the unknown traffic volumes with only a 10% probe vehicle penetration rate. The results validate the effectiveness and robustness of the proposed model and demonstrate its potential for practical applications.

Physics and Society,Machine Learning
A BPCA Based Missing Value Imputation and Its Impact on Traffic Incident Prediction

Huiping Li,Yinhai Wang,Meng Li

DOI: https://doi.org/10.1061/9780784481523.177

2018-01-01

Abstract:To improve the level of road safety, many traffic incident prediction models utilizing machine learning methods are proposed. However, traffic data applied to make a prediction is often not complete, and few studies are devoted to its potential impact on the prediction accuracy. In this study, several state-of-the-art machine learning methods like extreme gradient boosting, random forest, and support vector machine are adopted to make traffic incident predictions. 123 traffic incidents, and 5 months of microwave data on an urban expressway are collected. The missing pattern in our data is discussed and imputed by 3 methods: mean interpolation, probabilistic principal component analysis (PPCA), and Bayesian principal component analysis (BPCA). A sensitivity analysis is carried out under different missing rates. The numerical test revealed that BPCA performs slightly better than PPCA, but both produce higher and more stable prediction accuracy compared with mean interpolation, especially when ensemble learning techniques are adopted.
AGNP: Network-wide Short-Term Probabilistic Traffic Speed Prediction and Imputation

Meng Xu,Yining Di,Hongxing Ding,Zheng Zhu,Xiqun Chen,Hai Yang

DOI: https://doi.org/10.1016/j.commtr.2023.100099

2023-01-01

Communications in Transportation Research

Abstract:The data-driven Intelligent Transportation System (ITS) provides great support to travel decisions and system management but inevitably encounters the issue of data missing in monitoring systems. Hence, network-wide traffic state prediction and imputation is critical to recognizing the system level state of a transportation network. Abundant research works have adopted various approaches for traffic prediction and imputation. However, previous methods ignore the reliability analysis of the predicted/imputed traffic information. Thus, this study originally proposes an attentive graph neural process (AGNP) method for network-level short-term traffic speed prediction and imputation, simultaneously considering reliability. Firstly, the Gaussian process (GP) is used to model the observed traffic speed state. Such a stochastic process is further learned by the proposed AGNP method, which is utilized for inferring the congestion state on the remaining unobserved road segments. Data from a transportation network in Anhui Province, China, is used to conduct three experiments with increasing missing data ratio for model testing. Based on comparisons against other machine learning models, the results show that the proposed AGNP model can impute traffic networks and predict traffic speed with high-level per-formance. With the probabilistic confidence provided by the AGNP, reliability analysis is conducted both numerically and visually to show that the predicted distributions are beneficial to guide traffic control strategies and travel plans.
A Bipartite Graph Based Method for Traffic Continuous Data Imputation

Peng Jiaheng,Guan Tong,Liang Jun

DOI: https://doi.org/10.1109/cac59555.2023.10451480

2023-01-01

Abstract:Traffic information missing conditions are common in Intelligent Transportation System and would cause huge challenges for sequential traffic tasks, such as traffic speed prediction. Rare models are designed to tackle the long-range continuous data missing problems. Thus, in this paper, we propose a Bipartite Graph-based message passing Imputation Network (BGIN) to reconstruct traffic features data of nodes. In BGIN, the nodes are divided into two types: known nodes that contain traffic information and imputed nodes without data. First, a known information depth extraction module is given to achieve the messages passing among known nodes. Then, we construct a bipartite graph to represent the correlations between known and imputed nodes. Bipartite graph propagation module propagates the known information to imputed nodes. Finally, a imputation output module aggregates the messages to complete the imputation task. To evaluate the performance of our method, extensive experiments are conducted on a newly traffic imputation dataset called PeMS-impute. Experiment results demonstrate that BGIN achieves the best imputation effects than some graph based traffic prediction models and imputation models.
A Hybrid Data-Driven Framework for Spatiotemporal Traffic Flow Data Imputation

Peixiao Wang,Tao Hu,Fei Gao,Ruijie Wu,Wei Guo,Xinyan Zhu

DOI: https://doi.org/10.1109/jiot.2022.3151238

IF: 10.6

2022-01-01

IEEE Internet of Things Journal

Abstract:An accurate estimation of missing data in traffic flow is crucial in urban planning, intelligent transportation, economic geography, and other fields. Thus, improving the data quality of traffic flow is a necessary step in data modeling. Most existing studies use data-driven models to determine spatiotemporal patterns in traffic flow data and fill in the missing information automatically. However, simple data-driven models have unsatisfactory results for describing complex patterns in traffic flow and filling in missing data. This study treated the complex patterns in traffic flow as integrating multiple simple patterns and proposed a hybrid missing data imputation framework called ST-PTD. We used a specific time-series analysis to mine periodic patterns and proposed a novel matrix decomposition method to describe the trend of the traffic flow data. Furthermore, we fused the periodic and trend characteristics of the missing data using a novel dendritic neural network. We applied the framework in actual traffic flow data sets collected in Wuhan, China. The results showed that the ST-PTD framework outperformed the eight existing baselines in terms of imputation accuracy.

computer science, information systems,telecommunications,engineering, electrical & electronic
PCA-Based Missing Information Imputation for Real-Time Crash Likelihood Prediction under Imbalanced Data.

Jintao Ke,Shuaichao Zhang,Hai Yang,Xiqun (Michael) Chen

DOI: https://doi.org/10.1080/23249935.2018.1542414

2018-01-01

Transportmetrica A Transport Science

Abstract:As an important research topic, real-time crash likelihood prediction has been studied for many years. However, few research focuses on the missing data imputation in real-time crash likelihood prediction, although missing values are commonly observed due to breakdown of sensors or external interference. Besides, classifying imbalanced data is also a critical issue in real-time crash likelihood prediction, since the number of crash-prone cases is much smaller than that of non-crash cases. In this paper, three principal component analysis (PCA) based approaches are established for imputing missing values, while two kinds of solutions are developed to tackle the issue of imbalanced data. The results show that the proposed methods can help the classifiers achieve better predictive performance under situations with missing data. The two solutions, i.e. cost-sensitive learning, and synthetic minority oversampling technique (SMOTE), can help improve the sensitivity by adjusting the classifiers to pay more attention to the minority class.
A Spatiotemporal Approach for Traffic Data Imputation with Complicated Missing Patterns

Huiping Li,Meng Li,Xi Lin,Fang He,Yinhai Wang

DOI: https://doi.org/10.1016/j.trc.2020.102730

IF: 9.022

2020-01-01

Transportation Research Part C Emerging Technologies

Abstract:With the advent of intelligent transportation systems (ITS), spatiotemporal traffic data has gained growing importance in real-time monitoring, prediction, and control of traffic. However, in practical implementations, data collection devices are often faced with malfunctions caused by various unpredictable disruptions, thereby resulting in the so-called "missing value problems." In realistic cases, the disruptions to the data collection devices are often associated with some key events (e.g., power cut and natural disasters), in addition, along with other disruptions the missing value problem could be in a complicated manner with both randomly and completely missing patterns. To perform the imputation task with such complicated missing patterns, we propose a hybrid spatiotemporal method which utilizes the time series properties by "prophet" model and captures the spatial residuals information by iterative random forest model. The spatiotemporal method first applies the temporal part to fill the missing value and then adopts the spatial part to acquire the residual component of the missing values. The results of the two components are integrated into the final imputations. Based on the PeMS freeway dataset (PeMS, 2019) and an urban road dataset under extensive artificially designed scenarios like randomly, clustered non-completely and completely missing patterns, we test our proposed approach with some existing techniques such as K-Nearest Neighbor (KNN), Seasonal-Trend decomposition using Loess (STL), Bayesian tensor decomposition, Denoising AutoEncoder (DAE). The test results indicate that the hybrid method achieves the best imputation quality for most missing patterns, particularly for those with completely or hybrid missing patterns. Furthermore, the hybrid model still performs well under extreme missing rates as high as 0.9, which validates the robustness of the model in extreme situations.
A Latent-Factor-Model-Based Approach for Traffic Data Imputation with Road Network Information

Xing Su,Wenjie Sun,Chenting Song,Zhi Cai,Limin Guo

DOI: https://doi.org/10.3390/ijgi12090378

IF: 3.4

2023-09-16

ISPRS International Journal of Geo-Information

Abstract:With the rapid development of the economy, car ownership has grown rapidly, which causes many traffic problems. In recent years, intelligent transportation systems have been used to solve various traffic problems. To achieve effective and efficient traffic management, intelligent transportation systems need a large amount of complete traffic data. However, the current traffic data collection methods result in different forms of missing data. In the last twenty years, although many approaches have been proposed to impute missing data based on different mechanisms, these all have their limitations, which leads to low imputation accuracy, especially when the collected traffic data have a large amount of missing values. To this end, this paper proposes a latent-factor-model-based approach to impute the missing traffic data. In the proposed approach, the spatial information of the road network is first combined with the spatiotemporal matrix of the original traffic data. Then, the latent-factor-model-based algorithm is employed to impute the missing data in the combined matrix of the traffic data. Based on the real traffic data from METR-LA, we found that the imputation accuracy of the proposed approach was better than that of most of the current traffic-data-imputation approaches, especially when the original traffic data are limited.

geography, physical,remote sensing,computer science, information systems
The Combined Short-Term Forecasting Approach to Traffic Flow Based on Non-Parametric Regression

张晓利,贺国光,陆化普

DOI: https://doi.org/10.1061/40932(246)22

2009-01-01

Abstract:A combined method based on of Principal Component Analysis (PCA), Subtraction Fuzzy Clustering (SFC), and Non-Parametric Regression (NPR) is proposed in view of nonlinearity and uncertainty for short-term forecasting of real-time traffic flow. A highly efficient case database is created from the original traffic volumes after the operations of PCA and SFC. The data-driven method of K-nearest neighbours NPR is used to make the forecasting. An emulation experiment is designed to test the validity of the method. The example results show it is better than normal NPR and meets real-time requirement.
A Kriging based spatiotemporal approach for traffic volume data imputation

Hongtai Yang,Jianjiang Yang,Lee D Han,Xiaohan Liu,Li Pu,Shih-Miao Chin,Ho-Ling Hwang

DOI: https://doi.org/10.1371/journal.pone.0195957

IF: 3.7

2018-04-17

PLoS ONE

Abstract:Along with the rapid development of Intelligent Transportation Systems, traffic data collection technologies have progressed fast. The emergence of innovative data collection technologies such as remote traffic microwave sensor, Bluetooth sensor, GPS-based floating car method, and automated license plate recognition, has significantly increased the variety and volume of traffic data. Despite the development of these technologies, the missing data issue is still a problem that poses great challenge for data based applications such as traffic forecasting, real-time incident detection, dynamic route guidance, and massive evacuation optimization. A thorough literature review suggests most current imputation models either focus on the temporal nature of the traffic data and fail to consider the spatial information of neighboring locations or assume the data follow a certain distribution. These two issues reduce the imputation accuracy and limit the use of the corresponding imputation methods respectively. As a result, this paper presents a Kriging based data imputation approach that is able to fully utilize the spatiotemporal correlation in the traffic data and that does not assume the data follow any distribution. A set of scenarios with different missing rates are used to evaluate the performance of the proposed method. The performance of the proposed method was compared with that of two other widely used methods, historical average and K-nearest neighborhood. Comparison results indicate that the proposed method has the highest imputation accuracy and is more flexible compared to other methods.
An Experimental Evaluation of Imputation Models for Spatial-Temporal Traffic Data

Shengnan Guo,Tonglong Wei,Yiheng Huang,Miaomiao Zhao,Ran Chen,Yan Lin,Youfang Lin,Huaiyu Wan

2024-12-06

Abstract:Traffic data imputation is a critical preprocessing step in intelligent transportation systems, enabling advanced transportation services. Despite significant advancements in this field, selecting the most suitable model for practical applications remains challenging due to three key issues: 1) incomprehensive consideration of missing patterns that describe how data loss along spatial and temporal dimensions, 2) the lack of test on standardized datasets, and 3) insufficient evaluations. To this end, we first propose practice-oriented taxonomies for missing patterns and imputation models, systematically identifying all possible forms of real-world traffic data loss and analyzing the characteristics of existing models. Furthermore, we introduce a unified benchmarking pipeline to comprehensively evaluate 10 representative models across various missing patterns and rates. This work aims to provide a holistic understanding of traffic data imputation research and serve as a practical guideline.

Machine Learning
A hybrid model for missing traffic flow data imputation based on clustering and attention mechanism optimizing LSTM and AdaBoost

Qiang Shang,Yingping Tang,Longjiao Yin

DOI: https://doi.org/10.1038/s41598-024-77748-1

IF: 4.6

2024-11-03

Scientific Reports

Abstract:Reliable traffic flow data is not only crucial for traffic management and planning, but also the foundation for many intelligent applications. However, the phenomenon of missing traffic flow data often occurs, so we propose an imputation model for missing traffic flow data to overcome the randomness and instability bands of traffic flow. First, k-means clustering is used to classify road segments with traffic flow belonging to the same pattern into a group to utilize the spatial characteristics of roads fully. Then, the LSTM networks optimized with an attention mechanism are used as the base learner to extract the temporal dependence of the traffic flow. Finally, the AdaBoost algorithm is used to integrate all the LSTM-attention networks into a reinforced learner to impute the missing data. To validate the effectiveness of the proposed model, we use the PeMS dataset for validation, we impute the data with missing data rate from 10 to 60% under three missing modes, and we use multiple baseline models for comparison, which confirms that our proposed model improves the stability and accuracy of imputing the missing data of the traffic flow with different scenarios.

multidisciplinary sciences

PPCA-based missing data imputation for traffic flow volume

PPCA-Based Missing Data Imputation for Traffic Flow Volume: A Systematical Approach

Comparison on PPCA, KPPCA and MPPCA based missing data imputing for traffic flow

A Bpca Based Missing Value Imputing Method For Traffic Flow Volume Data

Missing Traffic Data: Comparison of Imputation Methods

Efficient missing data imputing for traffic flow by considering temporal and spatial dependence

Improving the Traffic Data Imputation Accuracy Using Temporal and Spatial Information

Missing data imputation for traffic flow based on improved local least squares

Estimating Traffic Flow in Large Road Networks Based on Multi-Source Traffic Data

A probabilistic model for missing traffic volume reconstruction based on data fusion

A BPCA Based Missing Value Imputation and Its Impact on Traffic Incident Prediction

AGNP: Network-wide Short-Term Probabilistic Traffic Speed Prediction and Imputation

A Bipartite Graph Based Method for Traffic Continuous Data Imputation

A Hybrid Data-Driven Framework for Spatiotemporal Traffic Flow Data Imputation

PCA-Based Missing Information Imputation for Real-Time Crash Likelihood Prediction under Imbalanced Data.

A Spatiotemporal Approach for Traffic Data Imputation with Complicated Missing Patterns

A Latent-Factor-Model-Based Approach for Traffic Data Imputation with Road Network Information

The Combined Short-Term Forecasting Approach to Traffic Flow Based on Non-Parametric Regression

A Kriging based spatiotemporal approach for traffic volume data imputation

An Experimental Evaluation of Imputation Models for Spatial-Temporal Traffic Data

A hybrid model for missing traffic flow data imputation based on clustering and attention mechanism optimizing LSTM and AdaBoost