Abstract:Advanced Driver Assistance Systems (ADAS) are designed with the main purpose of increasing the safety and comfort of vehicle occupants. Most of current computer vision-based ADAS perform detection and tracking tasks quite successfully under regular conditions, but are not completely reliable, particularly under adverse weather and changing lighting conditions, neither in complex situations with many overlapping objects. In this work we explore the use of hyperspectral imaging (HSI) in ADAS on the assumption that the distinct near infrared (NIR) spectral reflectances of different materials can help to better separate the objects in a driving scene. In particular, this paper describes some experimental results of the application of fully convolutional networks (FCN) to the image segmentation of HSI for ADAS applications. More specifically, our aim is to investigate to what extent the spatial features codified by convolutional filters can be helpful to improve the performance of HSI segmentation systems. With that aim, we use the HSI-Drive v1.1 dataset, which provides a set of labelled images recorded in real driving conditions with a small-size snapshot NIR-HSI camera. Finally, we analyze the implementability of such a HSI segmentation system by prototyping the developed FCN model together with the necessary hyperspectral cube preprocessing stage and characterizing its performance on an MPSoC.

What problem does this paper attempt to address?

### Problems the paper attempts to solve The paper aims to solve the problem of image segmentation in advanced driver - assistance systems (ADAS) in complex driving scenarios, especially in cases of bad weather and changing lighting conditions. Currently, computer - vision - based ADAS performs well under normal conditions, but is not completely reliable under these extreme conditions, especially in complex situations where there are many overlapping objects. Specifically, the paper explores the use of hyperspectral imaging (HSI) technology to improve the performance of ADAS. Hyperspectral imaging can better separate objects in driving scenarios by capturing the unique reflection characteristics of different materials in the near - infrared (NIR) band. The paper focuses in particular on the application of fully convolutional networks (FCN) in HSI image segmentation to evaluate the effect of spatial feature encoding on the performance improvement of the HSI segmentation system. ### Main research objectives 1. **Improve segmentation accuracy**: By combining hyperspectral information and spatial features extracted by convolution operations, improve the accuracy of object recognition and classification in complex driving scenarios. 2. **Cope with challenging conditions**: Especially in complex situations such as bad weather, lighting changes and object overlaps, ensure the robustness and reliability of the segmentation algorithm. 3. **Achieve real - time processing**: Achieve low - latency, high - performance HSI image segmentation on resource - constrained in - vehicle platforms to meet the real - time requirements of ADAS. ### Methods and datasets - **Dataset**: Use the HSI - Drive v1.1 dataset, which contains labeled images recorded under real - driving conditions and collected by a small - snapshot NIR - HSI camera. - **Model selection**: Adopt a fully convolutional network with the U - Net architecture, adjusted to adapt to the particularities of HSI data, and optimize the hyperparameters through grid search. - **Experimental setup**: Divide the dataset into training, validation and test subsets, and the metrics for evaluating segmentation performance include accuracy, precision and intersection - over - union (IoU). ### Conclusions Research shows that by introducing rich spectral and spatial information, the effect of HSI image segmentation can be significantly improved, especially showing better robustness and accuracy in complex driving scenarios. However, how to combine spectral and spatial information more effectively remains the focus of future research. In addition, the paper also demonstrates the feasibility of implementing HSI image segmentation on the Zynq UltraScale MPSoC platform, providing new ideas and technical support for future ADAS applications.

Exploring Fully Convolutional Networks for the Segmentation of Hyperspectral Imaging Applied to Advanced Driver Assistance Systems

On-chip Hyperspectral Image Segmentation with Fully Convolutional Networks for Scene Understanding in Autonomous Driving

Unifying Terrain Awareness Through Real-Time Semantic Segmentation

Hyperspectral Imaging-Based Perception in Autonomous Driving Scenarios: Benchmarking Baseline Semantic Segmentation Models

Rapid Deployment of Domain-specific Hyperspectral Image Processors with Application to Autonomous Driving

Using Channel-Wise Attention for Deep CNN Based Real-Time Semantic Segmentation With Class-Aware Edge Information

HSI-Drive v2.0: More Data for New Challenges in Scene Understanding for Autonomous Driving

A Brief Survey and an Application of Semantic Image Segmentation for Autonomous Driving

Vision-based Road Segmentation for Intelligent Vehicles using Deep Convolutional Neural Networks

Advanced Global Prototypical Segmentation Framework for Few-Shot Hyperspectral Image Classification

RGB and LiDAR Fusion-based 3D Semantic Segmentation for Autonomous Driving

Spectral Imagery Tensor Decomposition for Semantic Segmentation of Remote Sensing Data through Fully Convolutional Networks

Improving Semantic Segmentation Under Hazy Weather for Autonomous Vehicles Using Explainable Artificial Intelligence and Adaptive Dehazing Approach

Road Segmentation in SAR Satellite Images with Deep Fully-Convolutional Neural Networks

Learning Nighttime Semantic Segmentation the Hard Way

End to End Video Segmentation for Driving : Lane Detection For Autonomous Car

Unbalanced Class Learning Network With Scale-Adaptive Perception for Complicated Scene in Remote Sensing Images Segmentation

Hyperspectral Image Instance Segmentation Using Spectral–Spatial Feature Pyramid Network

SegHSI: Semantic Segmentation of Hyperspectral Images With Limited Labeled Pixels

SC-EADNet: A Self-supervised Contrastive Efficient Asymmetric Dilated Network for Hyperspectral Image Classification