Abstract:Object detection and semantic segmentation are two fundamental techniques for Intelligent Vehicles (IV) and Advanced Driving Assistance System (ADAS). Motivated by recent studies demonstrating that object detection and semantic segmentation are two highly-correlated tasks, this paper handles the problem of joint object detection and semantic segmentation in traffic scenes. Existing methods perform the joint object detection and semantic segmentation by sharing the same backbone network, but always ignore the interactive connection between the subdividing detection branch and segmentation branch, leading to the insufficient interaction between the two branches. Considering this situation, this paper proposes a joint object detection and semantic segmentation model with the cross-attention and inner-attention mechanisms. The cross-attention mechanism enables to build up the essential interaction between the subdividing detection branch and segmentation branch to fully make use of their correlation. In addition, the inner-attention contributes to strengthening the representations of feature maps in the model. Given an image, an encoder-decoder network is firstly used to extract initial feature maps. Then, the inner-attention mechanism is applied to strengthen the initial feature maps to obtain segmentation feature maps. Subsequently, the cross-attention mechanism utilizes the segmentation feature maps to guide the generation of object detection feature maps. Finally, the semantic segmentation is performed on the segmentation feature maps and object detection is performed on the detection feature maps. In the experiments, two well-known public traffic datasets are used to evaluate our model. Our model achieves the highest performance in comparison with several recently-proposed methods. In addition, some ablation studies are conducted to evaluate the proposed inner-attention and cross-attention mechanisms, and experiment results validate their effectiveness.

Joint Semantic-Instance Segmentation Method for Intelligent Transportation System

Associate Semantic-Instance Segmentation of 3D Point Clouds Based on Local Feature Extraction

PointMS: Semantic Segmentation for Point Cloud Based on Multi-scale Directional Convolution

Joint Semantic and Instance Segmentation in 3D Point Cloud Based on Transformer

A Multi-scale Network for Semantic Segmentation of 3D Point Clouds

Point Cloud Semantic Segmentation with Cross-Correction Features

Joint Point Clouds Semantic and Instance Segmentation by Local Aggregation and Clustering

A Joint Object Detection and Semantic Segmentation Model with Cross-Attention and Inner-Attention Mechanisms

Joint Semantic-Instance Segmentation of 3D Point Clouds: Instance Separation and Semantic Fusion.

High-Performance Feature Extraction Network for Point Cloud Semantic Segmentation

Multi‐stream Densely Connected Network for Semantic Segmentation

Semantic segmentation of large-scale point clouds based on dilated nearest neighbors graph

JSMNet Improving Indoor Point Cloud Semantic and Instance Segmentation through Self-Attention and Multiscale

A Network for Semantic Segmentation of Point Cloud Reorganized by Sectoral Cylindrical Partition

Extraction of local structure information of point clouds through space-filling curve for semantic segmentation

Robust 3D Semantic Segmentation Method Based on Multi-Modal Collaborative Learning

JSPNet: Learning Joint Semantic & Instance Segmentation of Point Clouds Via Feature Self-Similarity and Cross-Task Probability

Multi-Scale Superpoint Network for 3D Point Cloud Semantic Segmentation.

MSIDA-Net: Point Cloud Semantic Segmentation via Multi-Spatial Information and Dual Adaptive Blocks

SIESEF-FusionNet: Spatial Inter-correlation Enhancement and Spatially-Embedded Feature Fusion Network for LiDAR Point Cloud Semantic Segmentation

Accurate 3D Semantic Segmentation of Point Clouds for Intelligent Vehicles Based on Multi-view Edge Guidance and Fusion