Abstract:For autonomous vehicles, free-space detection is an essential part of visual perception. With the development of multi-modal convolutional neural networks (CNNs) in recent years, the performance of driving scene semantic segmentation algorithms has been dramatically improved. Therefore most free-space detection algorithms are developed based on multiple sensors. However, multi-modal CNNs have high data throughput and contain a large number of computationally intensive convolution calculations, limiting their feasibility for real-time applications. Field Programmable Gate Arrays (FPGAs) provide a unique combination of flexibility, performance, and low power for these problems to accommodate multi-modal data and the computational acceleration of different compression algorithms. Network lightweight methods offer great assurance for facilitating the deployment of CNNs on such resource-constrained devices. In this paper, we propose a network lightweight method for a multi-modal free-space detection algorithm. We first propose an FPGA-friendly multi-modal free-space detection lightweight network. It comprises operators that FPGA prefers and achieves a 95.54% MaxF score on the test set of KITTI-Road free-space detection tasks and 81 ms runtime when running on 700 W GPU devices. Then we present a pruning approach for this network to reduce the number of parameters in case the complete model exceeds the FPGA chip memory. The pruning is in two parts. For the feature extractors, we propose a data-dependent filter pruner according to the principle that the low-rank feature map contains less information. To not compromise the integrity of the multi-modal information, the pruner is independent for each modality. For the segmentation decoder, we apply a channel pruning approach to remove redundant parameters. Finally, we implement our designs on an FPGA board using 8-bit quantisation, and the accelerator achieves outstanding performance. A real-time application of scene segmentation on KITTI-Road is used to evaluate our algorithm, and the model achieves a 94.39% MaxF score and minimum 14 ms runtime on 20W FPGA devices.

Real-Time Object Detection and Semantic Segmentation Hardware System with Deep Learning Networks

Multi-task ADAS system on FPGA

A Scalable Real-time Semantic Segmentation Network for Autonomous Driving

An FPGA-Based Neural Network Overlay for ADAS Supporting Multi-Model and Multi-Mode

Design and implementation of FPGA-based deep learning object detection system

An Edge Computing System with AMD Xilinx FPGA AI Customer Platform for Advanced Driver Assistance System

An FPGA-Accelerated Design for Deep Learning Pedestrian Detection in Self-Driving Vehicles

An Efficient Real-Time Object Detection Framework on Resource-Constricted Hardware Devices via Software and Hardware Co-design

FPGA-Based Real-Time Object Detection and Classification System Using YOLO for Edge Computing

Algorithm-Hardware Co-Design of Single Shot Detector for Fast Object Detection on FPGAs

Real-time Joint Object Detection and Semantic Segmentation Network for Automated Driving

Using Channel-Wise Attention for Deep CNN Based Real-Time Semantic Segmentation With Class-Aware Edge Information

Driving Scene Perception Network: Real-time Joint Detection, Depth Estimation and Semantic Segmentation

Towards High-accuracy and Real-time Two-stage Small Object Detection on FPGA

Deep Learning Derived Object Detection and Tracking Technology Based on Sensor Fusion of Millimeter-Wave Radar/Video and Its Application on Embedded Systems

Real-time semantic segmentation on FPGAs for autonomous vehicles with hls4ml

Edge Real-Time Object Detection and DPU-Based Hardware Implementation for Optical Remote Sensing Images

EDSSA: An Encoder-Decoder Semantic Segmentation Networks Accelerator on OpenCL-Based FPGA Platform

Reconfigurable processor for deep learning in autonomous vehicles

FPGA-Based Vehicle Detection and Tracking Accelerator

FPGA-oriented lightweight multi-modal free-space detection network