Deep learning driven visual path prediction from a single image

Siyu Huang, Xi Li, Zhongfei Zhang, Zhouzhou He, Fei Wu, Wei Liu, Jinhui Tang, Yueting Zhuang

2016-09-26

Abstract:

What problem does this paper attempt to address?

Deep Learning Driven Visual Path Prediction from a Single Image.

Siyu Huang,Xi Li,Zhongfei (Mark) Zhang,Zhouzhou He,Fei Wu,Wei Liu,Jinhui Tang,Yueting Zhuang

DOI: https://doi.org/10.1109/TIP.2016.2613686

2016-01-01

Abstract:Capabilities of inference and prediction are the significant components of visual systems. Visual path prediction is an important and challenging task among them, with the goal to infer the future path of a visual object in a static scene. This task is complicated as it needs high-level semantic understandings of both the scenes and underlying motion patterns in video sequences. In practice, cluttered situations have also raised higher demands on the effectiveness and robustness of models. Motivated by these observations, we propose a deep learning framework, which simultaneously performs deep feature learning for visual representation in conjunction with spatiotemporal context modeling. After that, a unified path-planning scheme is proposed to make accurate path prediction based on the analytic results returned by the deep context models. The highly effective visual representation and deep context models ensure that our framework makes a deep semantic understanding of the scenes and motion patterns, consequently improving the performance on visual path prediction task. In experiments, we extensively evaluate the model’s performance by constructing two large benchmark datasets from the adaptation of video tracking datasets. The qualitative and quantitative experimental results show that our approach outperforms the state-of-the-art approaches and owns a better generalization capability.
Short-Term Forecasting of Traffic Flow and Speed: A Deep Learning Approach

Lingxiao Zhou,Xiqun (Michael) Chen

DOI: https://doi.org/10.1061/9780784481523.217

2018-01-01

Abstract:Real-time short-term traffic state prediction is essential for traffic management and is often conducted, based on historical data, at different traffic detection locations. However, the spatial and temporal dependencies of different detectors and different traffic state indicators need to be considered simultaneously, which makes short-term traffic prediction challenging. In this paper, we propose a deep learning based approach to simultaneously predict multi-step ahead traffic flow and speed for multiple detectors on a freeway. Based on the data collected by multiple loop detectors on the freeway, this paper transforms the time-space tensor into an image. The convolutional neural network (CNN) is employed to handle the multi-input multi-output problem. Compared to the support vector regression (SVR) and historical average (HA) models, the CNN-based model performs higher prediction accuracy and has a better performance in multistep traffic prediction. The proposed method can predict traffic speed and flow of several future time intervals within one prediction step, which saves a lot of prediction time.
One-Shot Learning of Visual Path Navigation for Autonomous Vehicles

Zhongying CuiZhu,Francois Charette,Amin Ghafourian,Debo Shi,Matthew Cui,Anjali Krishnamachar,Iman Soltani

DOI: https://doi.org/10.48550/arXiv.2306.08865

2023-06-15

Abstract:Autonomous driving presents many challenges due to the large number of scenarios the autonomous vehicle (AV) may encounter. End-to-end deep learning models are comparatively simplistic models that can handle a broad set of scenarios. However, end-to-end models require large amounts of diverse data to perform well. This paper presents a novel deep neural network that performs image-to-steering path navigation that helps with the data problem by adding one-shot learning to the system. Presented with a previously unseen path, the vehicle can drive the path autonomously after being shown the path once and without model retraining. In fact, the full path is not needed and images of the road junctions is sufficient. In-vehicle testing and offline testing are used to verify the performance of the proposed navigation and to compare different candidate architectures.

Computer Vision and Pattern Recognition,Machine Learning
Vehicle trajectory prediction in top-view image sequences based on deep learning method

Zahra Salahshoori Nejad,Hamed Heravi,Ali Rahimpour Jounghani,Abdollah Shahrezaie,Afshin Ebrahimi

DOI: https://doi.org/10.48550/arXiv.2102.01749

2021-02-02

Computer Vision and Pattern Recognition

Abstract:Annually, a large number of injuries and deaths around the world are related to motor vehicle accidents. This value has recently been reduced to some extent, via the use of driver-assistance systems. Developing driver-assistance systems (i.e., automated driving systems) can play a crucial role in reducing this number. Estimating and predicting surrounding vehicles' movement is essential for an automated vehicle and advanced safety systems. Moreover, predicting the trajectory is influenced by numerous factors, such as drivers' behavior during accidents, history of the vehicle's movement and the surrounding vehicles, and their position on the traffic scene. The vehicle must move over a safe path in traffic and react to other drivers' unpredictable behaviors in the shortest time. Herein, to predict automated vehicles' path, a model with low computational complexity is proposed, which is trained by images taken from the road's aerial image. Our method is based on an encoder-decoder model that utilizes a social tensor to model the effect of the surrounding vehicles' movement on the target vehicle. The proposed model can predict the vehicle's future path in any freeway only by viewing the images related to the history of the target vehicle's movement and its neighbors. Deep learning was used as a tool for extracting the features of these images. Using the HighD database, an image dataset of the road's aerial image was created, and the model's performance was evaluated on this new database. We achieved the RMSE of 1.91 for the next 5 seconds and found that the proposed method had less error than the best path-prediction methods in previous studies.
Depth Estimation of Traffic Scenes from Image Sequence Using Deep Learning.

Xiaoxu Liu,Wei Qi Yan

DOI: https://doi.org/10.1007/978-3-031-26431-3_15

2022-01-01

Abstract:Autonomous cars can accurately perceive the deployment of traffic scenes and the distance between visual objects in the scenarios through understanding the depth. Therefore, the depth estimation of scenes is a crucial step in the obstacle avoidance and pedestrian protection from autonomous vehicles. In this paper, a method for stereo depth estimation based on image sequences is introduced. In this project, we improve the performance of deep learning-based model by combining depth hints algorithm and MobileNetV2 encoder to enhance the loss function and increases computing speed. To the best of our knowledge, this is the first time MobileNetV2 is applied to depth estimation based on KITTI dataset.
Vehicle-Related Scene Understanding Using Deep Learning

Xiaoxu Liu,Minh Neuyen,Wei Qi Yan

DOI: https://doi.org/10.1007/978-981-15-3651-9_7

2019-01-01

Abstract:Automated driving is an inevitable trend in future transportation, it is also one of the eminent achievements in the matter of artificial intelligence. Deep learning produces a significant contribution to the progression of automatic driving. In this paper, our goal is to primarily deal with the issue of vehicle-related scene understanding using deep learning. To the best of our knowledge, this is the first time that we utilize our traffic environment as an object for scene understanding based on deep learning. Moreover, automatic scene segmentation and object detection are joined for traffic scene understanding. The techniques based on deep learning dramatically decrease human manipulations. Furthermore, the datasets in this paper consist of a large amount of our collected traffic images. Meanwhile, the performance of our algorithms is verified by the experiential results.
Deep Networks for Image-to-Image Prediction

Zhe Ren,Junchi Yan,Bin Liu,Xiaokang Yang,Hongyuan Zha

2021-01-01

Abstract:Convolutional neural networks (CNNs) have recently been very successful in a variety of computer vision tasks, especially on those linked to recognition. Optical flow estimation has not been among the tasks CNNs succeeded at. In this paper we construct CNNs which are capable of solving the optical flow estimation problem as a supervised learning task. We propose and compare two architectures: a generic architecture and another one including a layer that correlates feature vectors at different image locations. Since existing ground truth data sets are not sufficiently large to train a CNN, we generate a large synthetic Flying Chairs dataset. We show that networks trained on this unrealistic data still generalize very well to existing datasets such as Sintel and KITTI, achieving competitive accuracy at frame rates of 5 to 10 fps.
Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks

Henggang Cui,Vladan Radosavljevic,Fang-Chieh Chou,Tsung-Han Lin,Thi Nguyen,Tzu-Kuo Huang,Jeff Schneider,Nemanja Djuric

DOI: https://doi.org/10.48550/arXiv.1809.10732

2019-03-01

Abstract:Autonomous driving presents one of the largest problems that the robotics and artificial intelligence communities are facing at the moment, both in terms of difficulty and potential societal impact. Self-driving vehicles (SDVs) are expected to prevent road accidents and save millions of lives while improving the livelihood and life quality of many more. However, despite large interest and a number of industry players working in the autonomous domain, there still remains more to be done in order to develop a system capable of operating at a level comparable to best human drivers. One reason for this is high uncertainty of traffic behavior and large number of situations that an SDV may encounter on the roads, making it very difficult to create a fully generalizable system. To ensure safe and efficient operations, an autonomous vehicle is required to account for this uncertainty and to anticipate a multitude of possible behaviors of traffic actors in its surrounding. We address this critical problem and present a method to predict multiple possible trajectories of actors while also estimating their probabilities. The method encodes each actor's surrounding context into a raster image, used as input by deep convolutional networks to automatically derive relevant features for the task. Following extensive offline evaluation and comparison to state-of-the-art baselines, the method was successfully tested on SDVs in closed-course tests.

Robotics,Computer Vision and Pattern Recognition,Machine Learning
DDP: Diffusion Model for Dense Visual Prediction

Yuanfeng Ji,Zhe Chen,Enze Xie,Lanqing Hong,Xihui Liu,Zhaoqiang Liu,Tong Lu,Zhenguo Li,Ping Luo

DOI: https://doi.org/10.48550/arXiv.2303.17559

2023-03-30

Computer Vision and Pattern Recognition

Abstract:We propose a simple, efficient, yet powerful framework for dense visual predictions based on the conditional diffusion pipeline. Our approach follows a "noise-to-map" generative paradigm for prediction by progressively removing noise from a random Gaussian distribution, guided by the image. The method, called DDP, efficiently extends the denoising diffusion process into the modern perception pipeline. Without task-specific design and architecture customization, DDP is easy to generalize to most dense prediction tasks, e.g., semantic segmentation and depth estimation. In addition, DDP shows attractive properties such as dynamic inference and uncertainty awareness, in contrast to previous single-step discriminative methods. We show top results on three representative tasks with six diverse benchmarks, without tricks, DDP achieves state-of-the-art or competitive performance on each task compared to the specialist counterparts. For example, semantic segmentation (83.9 mIoU on Cityscapes), BEV map segmentation (70.6 mIoU on nuScenes), and depth estimation (0.05 REL on KITTI). We hope that our approach will serve as a solid baseline and facilitate future research
Visual Point Cloud Forecasting enables Scalable Autonomous Driving

Zetong Yang,Li Chen,Yanan Sun,Hongyang Li

DOI: https://doi.org/10.48550/arXiv.2312.17655

2023-12-29

Computer Vision and Pattern Recognition

Abstract:In contrast to extensive studies on general vision, pre-training for scalable visual autonomous driving remains seldom explored. Visual autonomous driving applications require features encompassing semantics, 3D geometry, and temporal information simultaneously for joint perception, prediction, and planning, posing dramatic challenges for pre-training. To resolve this, we bring up a new pre-training task termed as visual point cloud forecasting - predicting future point clouds from historical visual input. The key merit of this task captures the synergic learning of semantics, 3D structures, and temporal dynamics. Hence it shows superiority in various downstream tasks. To cope with this new problem, we present ViDAR, a general model to pre-train downstream visual encoders. It first extracts historical embeddings by the encoder. These representations are then transformed to 3D geometric space via a novel Latent Rendering operator for future point cloud prediction. Experiments show significant gain in downstream tasks, e.g., 3.1% NDS on 3D detection, ~10% error reduction on motion forecasting, and ~15% less collision rate on planning.
DeepTrack: Lightweight Deep Learning for Vehicle Path Prediction in Highways

Vinit Katariya,Mohammadreza Baharani,Nichole Morris,Omidreza Shoghli,Hamed Tabkhi

DOI: https://doi.org/10.1109/TITS.2022.3172015

2022-05-26

Abstract:Vehicle trajectory prediction is essential for enabling safety-critical intelligent transportation systems (ITS) applications used in management and operations. While there have been some promising advances in the field, there is a need for modern deep learning algorithms that allow real-time trajectory prediction on embedded IoT devices. This article presents DeepTrack, a novel deep learning algorithm customized for real-time vehicle trajectory prediction and monitoring applications in arterial management, freeway management, traffic incident management, and work zone management for high-speed incoming traffic. In contrast to previous methods, the vehicle dynamics are encoded using Temporal Convolutional Networks (TCNs) to provide more robust time prediction with less computation. DeepTrack also uses depthwise convolution, which reduces the complexity of models compared to existing approaches in terms of model size and operations. Overall, our experimental results demonstrate that DeepTrack achieves comparable accuracy to state-of-the-art trajectory prediction models but with smaller model sizes and lower computational complexity, making it more suitable for real-world deployment.

Machine Learning,Robotics
Self-Driving Car Steering Angle Prediction Based on Image Recognition

Shuyang Du,Haoli Guo,Andrew Simpson

DOI: https://doi.org/10.48550/arXiv.1912.05440

2019-12-11

Computer Vision and Pattern Recognition

Abstract:Self-driving vehicles have expanded dramatically over the last few years. Udacity has release a dataset containing, among other data, a set of images with the steering angle captured during driving. The Udacity challenge aimed to predict steering angle based on only the provided images. We explore two different models to perform high quality prediction of steering angles based on images using different deep learning techniques including Transfer Learning, 3D CNN, LSTM and ResNet. If the Udacity challenge was still ongoing, both of our models would have placed in the top ten of all entries.
Pedestrian Motion Trajectory Prediction in Intelligent Driving from Far Shot First-Person Perspective Video

Yingfeng Cai,Lei Dai,Hai Wang,Long Chen,Yicheng Li,Miguel Angel Sotelo,Zhixiong Li

DOI: https://doi.org/10.1109/tits.2021.3052908

IF: 8.5

2021-01-01

IEEE Transactions on Intelligent Transportation Systems

Abstract:Pedestrian motion trajectory prediction is an important task in intelligent driving, and it can provide a valuable reference for the subsequent path decision of intelligent driving. However, so far, there are only a few models in the field of specific pedestrian motion track prediction in intelligent driving from far shot first-person perspective video. To accomplish this task, we proposed a deep learning model for pedestrian motion trajectory prediction from far shot first-person perspective video with four key innovations: a) A macroscopic pedestrian trajectory prediction module is established under the close correlation between neighboring frames to estimate the pedestrian motion track on the whole; b) A relative motion transformation module of vehicle-mounted camera is designed to consider the effect of vehicle-mounted camera's ego-motion on the pedestrian motion track; c) We set up a circular training module to maintain the number of parameters in our model to simplify and reduce the size of model; d) A new far shot first-person pedestrian motion dataset under intelligent driving is specifically established to train and test the proposed model. The above four modules are integrated into the proposed deep learning model, which achieves state-of-the-art results for predicting pedestrian motion trajectory from both far and close shot first-person perspective video.

engineering, electrical & electronic,transportation science & technology, civil
ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries

Junru Gu,Chenxu Hu,Tianyuan Zhang,Xuanyao Chen,Yilun Wang,Yue Wang,Hang Zhao

2023-06-19

Abstract:Perception and prediction are two separate modules in the existing autonomous driving systems. They interact with each other via hand-picked features such as agent bounding boxes and trajectories. Due to this separation, prediction, as a downstream module, only receives limited information from the perception module. To make matters worse, errors from the perception modules can propagate and accumulate, adversely affecting the prediction results. In this work, we propose ViP3D, a query-based visual trajectory prediction pipeline that exploits rich information from raw videos to directly predict future trajectories of agents in a scene. ViP3D employs sparse agent queries to detect, track, and predict throughout the pipeline, making it the first fully differentiable vision-based trajectory prediction approach. Instead of using historical feature maps and trajectories, useful information from previous timestamps is encoded in agent queries, which makes ViP3D a concise streaming prediction method. Furthermore, extensive experimental results on the nuScenes dataset show the strong vision-based prediction performance of ViP3D over traditional pipelines and previous end-to-end models.

Computer Vision and Pattern Recognition,Robotics
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene from Sparse LiDAR Data and Single Color Image

Jiaxiong Qiu,Zhaopeng Cui,Yinda Zhang,Xingdi Zhang,Shuaicheng Liu,Bing Zeng,Marc Pollefeys

DOI: https://doi.org/10.1109/cvpr.2019.00343

2018-01-01

Abstract:In this paper, we propose a deep learning architecture that produces accurate dense depth for the outdoor scene from a single color image and a sparse depth. Inspired by the indoor depth completion, our network estimates surface normals as the intermediate representation to produce dense depth, and can be trained end-to-end. With a modified encoder-decoder structure, our network effectively fuses the dense color image and the sparse LiDAR depth. To address outdoor specific challenges, our network predicts a confidence mask to handle mixed LiDAR signals near foreground boundaries due to occlusion, and combines estimates from the color image and surface normals with learned attention maps to improve the depth accuracy especially for distant areas. Extensive experiments demonstrate that our model improves upon the state-of-the-art performance on KITTI depth completion benchmark. Ablation study shows the positive impact of each model components to the final performance, and comprehensive analysis shows that our model generalizes well to the input with higher sparsity or from indoor scenes.
Multimodal Trajectory Prediction for Autonomous Driving on Unstructured Roads using Deep Convolutional Network

Lei Li,Zhifa Chen,Jian Wang,Bin Zhou,Guizhen Yu,Xiaoxuan Chen

2024-09-27

Abstract:Recently, the application of autonomous driving in open-pit mining has garnered increasing attention for achieving safe and efficient mineral transportation. Compared to urban structured roads, unstructured roads in mining sites have uneven boundaries and lack clearly defined lane markings. This leads to a lack of sufficient constraint information for predicting the trajectories of other human-driven vehicles, resulting in higher uncertainty in trajectory prediction problems. A method is proposed to predict multiple possible trajectories and their probabilities of the target vehicle. The surrounding environment and historical trajectories of the target vehicle are encoded as a rasterized image, which is used as input to our deep convolutional network to predict the target vehicle's multiple possible trajectories. The method underwent offline testing on a dataset specifically designed for autonomous driving scenarios in open-pit mining and was compared and evaluated against physics-based method. The open-source code and data are available at <a class="link-external link-https" href="https://github.com/LLsxyc/mine_motion_prediction.git" rel="external noopener nofollow">this https URL</a>

Artificial Intelligence
FPPN: Future Pseudo-LiDAR Frame Prediction for Autonomous Driving

Xudong Huang,Chunyu Lin,Haojie Liu,Lang Nie,Yao Zhao

DOI: https://doi.org/10.48550/arXiv.2112.04401

2021-12-08

Computer Vision and Pattern Recognition

Abstract:LiDAR sensors are widely used in autonomous driving due to the reliable 3D spatial information. However, the data of LiDAR is sparse and the frequency of LiDAR is lower than that of cameras. To generate denser point clouds spatially and temporally, we propose the first future pseudo-LiDAR frame prediction network. Given the consecutive sparse depth maps and RGB images, we first predict a future dense depth map based on dynamic motion information coarsely. To eliminate the errors of optical flow estimation, an inter-frame aggregation module is proposed to fuse the warped depth maps with adaptive weights. Then, we refine the predicted dense depth map using static contextual information. The future pseudo-LiDAR frame can be obtained by converting the predicted dense depth map into corresponding 3D point clouds. Experimental results show that our method outperforms the existing solutions on the popular KITTI benchmark.
Prediction of the driver's focus of attention based on feature visualization of a deep autonomous driving model

Tao Huang,Rui Fu

DOI: https://doi.org/10.1016/j.knosys.2022.109006

2022-05-19

Abstract:Driver focus of attention (DFoA) is a fundamental research problem in human-like autonomous driving systems. However, most existing methods require large amounts of ground-truth DFoA data for training, which are difficult to collect. Inspired by the visual interpretability of neural networks, this study innovatively developed a DFoA prediction method based on the feature visualization of a deep autonomous driving model that does not require ground-truth DFoA data to train. Here, we propose a multimodal spatiotemporal convolutional network with an attention mechanism for DFoA prediction. First, semantic and depth images were generated from RGB video frames to enable the multipath convolutional network to learn the spatiotemporal information of successive images. A parameter-free attention mechanism with 3-D weights was incorporated as an energy function to calculate the importance of each neuron. A graph attention network was used to learn the most driving behavior-relevant semantic context features. The learned features were fused, and a convolutional long short-term memory network (ConvLSTM) was adopted to achieve the evolution of the fused features in successive frames while considering historical scene variation. Finally, a novel feature visualization method was designed to predict the DFoA by visualizing the driving behavior-relevant features. The experimental results demonstrated that the proposed method can accurately predict DFoA.

computer science, artificial intelligence
Traffic Speed Prediction for Urban Transportation Network: A Path Based Deep Learning Approach

Jiawei Wang,Ruixiang Chen,Zhaocheng He

DOI: https://doi.org/10.1016/j.trc.2019.02.002

IF: 9.022

2019-01-01

Transportation Research Part C Emerging Technologies

Abstract:Traffic prediction, as an important part of intelligent transportation systems, plays a critical role in traffic state monitoring. While many studies accomplished traffic forecasting task with deep learning models, there is still an open issue of exploiting spatial-temporal traffic state features for better prediction performance, and the model interpretability has not been taken serious. In this study, we propose a path based deep learning framework which can produce better traffic speed prediction at a city wide scale, furthermore, the model is both rational and interpretable in the context of urban transportation. Specifically, we divide the road network into critical paths, which is helpful to mine the traffic flow mechanism. Then, each critical path is modeled through the bidirectional long short-term memory neural network (Bi-LSTM NN), and multiple Bi-LSTM layers are stacked to incorporate temporal information. At the stage of traffic prediction, the spatial-temporal features captured from these processes are fed into a fully-connected layer. Finally, results for each path are ensembled for network-wise traffic speed prediction. In the empirical studies, we compare the proposed model with multiple benchmark methods. Under a series of prediction scenarios (i.e., different input and prediction horizons), the superior performance of the proposed framework is validated. Moreover, by analyzing feature from hidden layer output, the study explains the physical meaning of the hidden feature and illustrate model's interpretability.
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network

David Eigen,Christian Puhrsch,Rob Fergus

DOI: https://doi.org/10.48550/arXiv.1406.2283

2014-06-09

Computer Vision and Pattern Recognition

Abstract:Predicting depth is an essential component in understanding the 3D geometry of a scene. While for stereo images local correspondence suffices for estimation, finding depth relations from a single image is less straightforward, requiring integration of both global and local information from various cues. Moreover, the task is inherently ambiguous, with a large source of uncertainty coming from the overall scale. In this paper, we present a new method that addresses this task by employing two deep network stacks: one that makes a coarse global prediction based on the entire image, and another that refines this prediction locally. We also apply a scale-invariant error to help measure depth relations rather than scale. By leveraging the raw datasets as large sources of training data, our method achieves state-of-the-art results on both NYU Depth and KITTI, and matches detailed depth boundaries without the need for superpixelation.

Deep learning driven visual path prediction from a single image

Deep Learning Driven Visual Path Prediction from a Single Image.

Short-Term Forecasting of Traffic Flow and Speed: A Deep Learning Approach

One-Shot Learning of Visual Path Navigation for Autonomous Vehicles

Vehicle trajectory prediction in top-view image sequences based on deep learning method

Depth Estimation of Traffic Scenes from Image Sequence Using Deep Learning.

Vehicle-Related Scene Understanding Using Deep Learning

Deep Networks for Image-to-Image Prediction

Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks

DDP: Diffusion Model for Dense Visual Prediction

Visual Point Cloud Forecasting enables Scalable Autonomous Driving

DeepTrack: Lightweight Deep Learning for Vehicle Path Prediction in Highways

Self-Driving Car Steering Angle Prediction Based on Image Recognition

Pedestrian Motion Trajectory Prediction in Intelligent Driving from Far Shot First-Person Perspective Video

ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries

DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene from Sparse LiDAR Data and Single Color Image

Multimodal Trajectory Prediction for Autonomous Driving on Unstructured Roads using Deep Convolutional Network

FPPN: Future Pseudo-LiDAR Frame Prediction for Autonomous Driving

Prediction of the driver's focus of attention based on feature visualization of a deep autonomous driving model

Traffic Speed Prediction for Urban Transportation Network: A Path Based Deep Learning Approach

Depth Map Prediction from a Single Image using a Multi-Scale Deep Network