Abstract:For autonomous vehicles, free-space detection is an essential part of visual perception. With the development of multi-modal convolutional neural networks (CNNs) in recent years, the performance of driving scene semantic segmentation algorithms has been dramatically improved. Therefore most free-space detection algorithms are developed based on multiple sensors. However, multi-modal CNNs have high data throughput and contain a large number of computationally intensive convolution calculations, limiting their feasibility for real-time applications. Field Programmable Gate Arrays (FPGAs) provide a unique combination of flexibility, performance, and low power for these problems to accommodate multi-modal data and the computational acceleration of different compression algorithms. Network lightweight methods offer great assurance for facilitating the deployment of CNNs on such resource-constrained devices. In this paper, we propose a network lightweight method for a multi-modal free-space detection algorithm. We first propose an FPGA-friendly multi-modal free-space detection lightweight network. It comprises operators that FPGA prefers and achieves a 95.54% MaxF score on the test set of KITTI-Road free-space detection tasks and 81 ms runtime when running on 700 W GPU devices. Then we present a pruning approach for this network to reduce the number of parameters in case the complete model exceeds the FPGA chip memory. The pruning is in two parts. For the feature extractors, we propose a data-dependent filter pruner according to the principle that the low-rank feature map contains less information. To not compromise the integrity of the multi-modal information, the pruner is independent for each modality. For the segmentation decoder, we apply a channel pruning approach to remove redundant parameters. Finally, we implement our designs on an FPGA board using 8-bit quantisation, and the accelerator achieves outstanding performance. A real-time application of scene segmentation on KITTI-Road is used to evaluate our algorithm, and the model achieves a 94.39% MaxF score and minimum 14 ms runtime on 20W FPGA devices.

Color Segmentation on FPGA Using Minimum Distance Classifier for Automatic Road Sign Detection

Design and Implementation of Visual Processing System Based on FPGA

Multi-objects Real Time Recognition Based on Color Information

High-performance Subpixel Edge Location Based on FPGA for Horizon Sensors

FPGA Implementation of Feature Detection Algorithm Based on High Level Synthesis

An FPGA-Based Hardware Accelerator for Traffic Sign Detection

Road Traffic Sign Detection and Recognition using Adaptive Color Segmentation and Deep Learning

High-Speed Color Sorting Algorithm Based on FPGA Implementation.

Exploiting Low-level Representations for Ultra-Fast Road Segmentation

FPGA-Based Real-Time Object Detection and Classification System Using YOLO for Edge Computing

Benchmarking Deep Learning Frameworks and Investigating FPGA Deployment for Traffic Sign Classification and Detection

A Real World Detection System - Combining Color, Shape and Appearance to Enable Real-time Road Sign Detection.

A High-Speed Low-Cost VLSI System Capable of On-Chip Online Learning for Dynamic Vision Sensor Data Classification

Traffic signs detection and recognition systems by light-weight multi-stage network

FPGA-oriented lightweight multi-modal free-space detection network

A Digital Fuzzy Edge Detector for Color Images

An FPGA-based Parallel Architecture for Face Detection using Mixed Color Models

A Fusion Approach to Detect Traffic Signs Using Registered Color Images and Noisy Airborne LiDAR Data

A CNN-based Traffic Sign Detection and Classification Method Using Priori Knowledge

An Improved Method for Red Segmentation Based Traffic Sign Detection