Abstract:Fully self-driving (FSD) vehicles are becoming increasing popular over the last few years and companies are investing significantly into its research and development. In the recent years, FSD technology innovators like Tesla, Google etc. have been working on proprietary autonomous driving stacks and have been able to successfully bring the vehicle to the roads. On the other end, organizations like Autoware Foundation and Baidu are fueling the growth of self-driving mobility using open source stacks. These organizations firmly believe in enabling autonomous driving technology for everyone and support developing software stacks through the open source community that is SoC vendor agnostic. In this proposed solution we describe a vision control unit for a fully self-driving vehicle developed on Xilinx MPSoC platform using open source software components. The vision control unit of an FSD vehicle is responsible for camera video capture, image processing and rendering, AI algorithm processing, data and meta-data transfer to next stage of the FSD pipeline. In this proposed solution we have used many open source stacks and frameworks for video and AI processing. The processing of the video pipeline and algorithms take full advantage of the pipelining and parallelism using all the heterogenous cores of the Xilinx MPSoC. In addition, we have developed an extensible, scalable, adaptable and configurable AI backend framework, XTA, for acceleration purposes that is derived from a popular, open source AI backend framework, TVM-VTA. XTA uses all the MPSoC cores for its computation in a parallel and pipelined fashion. XTA also adapts to the compute and memory parameters of the system and can scale to achieve optimal performance for any given AI problem. The FSD system design is based on a distributed system architecture and uses open source components like Autoware for autonomous driving algorithms, ROS and Distributed Data Services as a messaging middleware between the functional nodes and a real-time kernel to coordinate the actions. The details of image capture, rendering and AI processing of the vision perception pipeline will be presented along with the performance measurements of the vision pipeline. In this proposed solution we will demonstrate some of the key use cases of vision perception unit like surround vision and object detection. In addition, we will also show the capability of Xilinx MPSoC technology to handle multiple channels of real time camera and the integration with the Lidar/Radar point cloud data to feed into the decision-making unit of the overall system. The system is also designed with the capability to update the vision control unit through Over the Air Update (OTA). It is also envisioned that the core AI engine will require regular updates with the latest training values; hence a built-in platform level mechanism supporting such capability is essential for real world deployment.

An FPGA-based Low Latency Sensing and Communication Platform for Collaborative Autonomous Driving

Vehicular Edge Computing for Multi-Vehicle Perception

Slim-FCP: Lightweight-Feature-Based Cooperative Perception for Connected Automated Vehicles

Live Demonstration: An Integrated Computing and Communication Platform for Vehicle-Infrastructure Cooperative Autonomous Driving

Design and Implementation of Driverless Perceptual System Based on CPU + FPGA

Accelerating autonomous vehicles: Harnessing FPGA power for deep learning advancements

EdgeCooper: Network-Aware Cooperative LiDAR Perception for Enhanced Vehicular Awareness

Co-Vast: A Simulation Platform for Vehicle Collaboration in Autonomous Driving Scenarios

VIPS: Real-Time Perception Fusion for Infrastructure-Assisted Autonomous Driving

Towards Collaborative Autonomous Driving: Simulation Platform and End-to-End System

Real-time Vehicle-to-Vehicle Communication Based Network Cooperative Control System through Distributed Database and Multimodal Perception: Demonstrated in Crossroads

A New Vehicular Fog Computing Architecture for Cooperative Sensing of Autonomous Driving

Panoptic Driving Perception Model and Inference Acceleration Based on FPGA

Adaptive Communications in Collaborative Perception with Domain Alignment for Autonomous Driving

A network-level connected autonomous driving evaluation platform implementing C-V2X technology

Implementation of a perception system for autonomous vehicles using a detection-segmentation network in SoC FPGA

Hybrid Sensing Data Fusion of Cooperative Perception for Autonomous Driving with Augmented Vehicular Reality

Implementation of In-vehicle Multi-sensor Information Fusion Gateway for Cooperative Driving

Toward Collaborative Occlusion-free Perception in Connected Autonomous Vehicles

Vision Control Unit in Fully Self Driving Vehicles using Xilinx MPSoC and Opensource Stack

Real Time Object Detection Based on FPGA with Big Data