Abstract:Object detection and semantic segmentation are two fundamental techniques for Intelligent Vehicles (IV) and Advanced Driving Assistance System (ADAS). Motivated by recent studies demonstrating that object detection and semantic segmentation are two highly-correlated tasks, this paper handles the problem of joint object detection and semantic segmentation in traffic scenes. Existing methods perform the joint object detection and semantic segmentation by sharing the same backbone network, but always ignore the interactive connection between the subdividing detection branch and segmentation branch, leading to the insufficient interaction between the two branches. Considering this situation, this paper proposes a joint object detection and semantic segmentation model with the cross-attention and inner-attention mechanisms. The cross-attention mechanism enables to build up the essential interaction between the subdividing detection branch and segmentation branch to fully make use of their correlation. In addition, the inner-attention contributes to strengthening the representations of feature maps in the model. Given an image, an encoder-decoder network is firstly used to extract initial feature maps. Then, the inner-attention mechanism is applied to strengthen the initial feature maps to obtain segmentation feature maps. Subsequently, the cross-attention mechanism utilizes the segmentation feature maps to guide the generation of object detection feature maps. Finally, the semantic segmentation is performed on the segmentation feature maps and object detection is performed on the detection feature maps. In the experiments, two well-known public traffic datasets are used to evaluate our model. Our model achieves the highest performance in comparison with several recently-proposed methods. In addition, some ablation studies are conducted to evaluate the proposed inner-attention and cross-attention mechanisms, and experiment results validate their effectiveness.

Vehicle Motion Segmentation Via Combining Neural Networks and Geometric Methods

From One To Many: Unsupervised Traversable Area Segmentation In Off-Road Environment

Moving vehicle tracking and scene understanding: A hybrid approach

Zero-Shot Monocular Motion Segmentation in the Wild by Combining Deep Learning with Geometric Motion Model Fusion

A Multi-view 3D Vehicle Detection Method Based On Novel 3D Proposal Generation Method

Vehicle Instance Segmentation from Aerial Image and Video Using a Multi-Task Learning Residual Fully Convolutional Network

A Joint Object Detection and Semantic Segmentation Model with Cross-Attention and Inner-Attention Mechanisms

Vehicle Re-Identification and Tracking Based on Video Segmentation.

Moving Object Segmentation in 3D LiDAR Data: A Learning-based Approach Exploiting Sequential Data

Vehicle Instance Segmentation From Aerial Image and Video Using a Multitask Learning Residual Fully Convolutional Network

Efficient Spatial-Temporal Information Fusion for LiDAR-Based 3D Moving Object Segmentation

Multi-Motion Segmentation: Combining Geometric Model-Fitting and Optical Flow for RGB Sensors

Moving Vehicles Segmentation Based On Gaussian Motion Model

Vehicle Segmentation with Coarse Distance Estimation Based on Monocular Vision

Scene-Adaptive 3-D Semantic Segmentation Method Based on Multiphase Edge Enhancement for Intelligent Vehicles

Robust 3D Semantic Segmentation Method Based on Multi-Modal Collaborative Learning

Motion Segmentation from a Moving Monocular Camera

Event-Free Moving Object Segmentation from Moving Ego Vehicle

A Sim2Real Deep Learning Approach for the Transformation of Images from Multiple Vehicle-Mounted Cameras to a Semantically Segmented Image in Bird's Eye View

OE-BevSeg: An Object Informed and Environment Aware Multimodal Framework for Bird's-eye-view Vehicle Semantic Segmentation

Multitarget Vehicle Tracking and Motion State Estimation Using a Novel Driving Environment Perception System of Intelligent Vehicles