Abstract:Owing to rapid developments in driverless technologies, vision tasks for unmanned vehicles have gained considerable attention, particularly in multimodal-based urban scene parsing. Although deep-learning algorithms have outperformed traditional models in such tasks, they cannot operate on mobile devices and edge networks owing to the coarse-grained cross-modal complementary information alignment, inadequate modeling of semantic-category relations, overabundance of parameters, and high computational complexity. To address these issues, a multilayer guided semantic graph network via knowledge distillation (MGSGNet-S*) is proposed for red-green-blue-thermal urban scene parsing. First, a new cross-modal adaptive fusion module adjusts pixel-level adaptive modal complementary information by incorporating additional deep modal information and residual cross-modal matrix fine-grained attention. Second, a novel semantic graph module overcomes the misclassification problems of objects of the same semantic class during low-level encoding by incorporating high-level information in the Euclidean space and modeling semantic graph relationships in the non-Euclidean space. Finally, to strike the balance between accuracy and efficiency, a tailored framework optimally utilizes effective knowledge of pixel intra- and inter-class similarity, fusion features, and cross-modal correlation. Experimental results indicate that MGSGNet-S* considerably outperforms relevant state-of-the-art methods with fewer parameters and lower computational costs. The numbers of parameters and floating-point operations were reduced by 95.69% and 93.34%, respectively, relative to those for the teacher model, thus demonstrating stronger inferencing capabilities at 28.65 frames per second. The source codes and results are available at https://github.com/Tortoisewhp/MGSGNet.

Lightweight cross-guided contextual perceptive network for visible–infrared urban road scene parsing

Context-Integrated and Feature-Refined Network for Lightweight Object Parsing

FoveaNet: Perspective-Aware Urban Scene Parsing

Efficient Light Deep Network for Street Scene Parsing.

Adaptive Context Network for Scene Parsing

ECNet: an Efficient and Context-Aware Network for Street Scene Parsing.

Real-time Semantic Segmentation in Traffic Scene Using Cross Stage Partial-based Encoder–decoder Network

Global-Guided Selective Context Network for Scene Parsing

Context Prior Based Semantic-Spatial Graph Network for Human Parsing.

EFRNet: Efficient Feature Reconstructing Network for Real-Time Scene Parsing

MGSGNet-S*: Multilayer Guided Semantic Graph Network Via Knowledge Distillation for RGB-Thermal Urban Scene Parsing

Fully Combined Convolutional Network With Soft Cost Function For Traffic Scene Parsing

Multifeature Selective Fusion Network for Real-Time Driving Scene Parsing

C2Net: Road Extraction via Context Perception and Cross Spatial-Scale Feature Interaction

Local and Global Context Attentive Fusion Network for Traffic Scene Parsing

MCFNet: Multi-Attentional Class Feature Augmentation Network for Real-Time Scene Parsing

Road and Car Extraction Using UAV Images Via Efficient Dual Contextual Parsing Network

Toward Achieving Robust Low-Level and High-Level Scene Parsing

NDNet: Spacewise Multiscale Representation Learning via Neighbor Decoupling for Real-Time Driving Scene Parsing

RAPNet: Residual Atrous Pyramid Network for Importance-Aware Street Scene Parsing