Abstract:Low-level details and high-level semantics are both essential to the semantic segmentation task. However, to speed up the model inference, current approaches almost always sacrifice the low-level details, leading to a considerable decrease in accuracy. We propose to treat these spatial details and categorical semantics separately to achieve high accuracy and high efficiency for real-time semantic segmentation. For this purpose, we propose an efficient and effective architecture with a good trade-off between speed and accuracy, termed Bilateral Segmentation Network (BiSeNet V2). This architecture involves the following: (i) A detail branch, with wide channels and shallow layers to capture low-level details and generate high-resolution feature representation; (ii) A semantics branch, with narrow channels and deep layers to obtain high-level semantic context. The detail branch has wide channel dimensions and shallow layers, while the semantics branch has narrow channel dimensions and deep layers. Due to the reduction in the channel capacity and the use of a fast-downsampling strategy, the semantics branch is lightweight and can be implemented by any efficient model. We design a guided aggregation layer to enhance mutual connections and fuse both types of feature representation. Moreover, a booster training strategy is designed to improve the segmentation performance without any extra inference cost. Extensive quantitative and qualitative evaluations demonstrate that the proposed architecture shows favorable performance compared to several state-of-the-art real-time semantic segmentation approaches. Specifically, for a input, we achieve 72.6% Mean IoU on the Cityscapes test set with a speed of 156 FPS on one NVIDIA GeForce GTX 1080 Ti card, which is significantly faster than existing methods, yet we achieve better segmentation accuracy. The code and trained models are available online at https://git.io/BiSeNet.

Purify then Guide: A Bi-directional Bridge Network for Open-Vocabulary Semantic Segmentation

Open-Vocabulary Semantic Segmentation with Image Embedding Balancing

Learning Open-vocabulary Semantic Segmentation Models from Natural Language Supervision.

A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model

Open-Vocabulary Segmentation with Semantic-Assisted Calibration

BiSeNet V2: Bilateral Network with Guided Aggregation for Real-Time Semantic Segmentation

SAN: Side Adapter Network for Open-vocabulary Semantic Segmentation

Attention Guided Global Enhancement and Local Refinement Network for Semantic Segmentation

Global Knowledge Calibration for Fast Open-Vocabulary Segmentation

Bi-directional Attention Feature Enhancement for Video Instance Segmentation.

A Deep Semantic Segmentation Network with Semantic and Contextual Refinements

Instance Brownian Bridge as Texts for Open-vocabulary Video Instance Segmentation

Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentation

Side Adapter Network for Open-Vocabulary Semantic Segmentation

Transformer Meets Convolution: A Bilateral Awareness Network for Semantic Segmentation of Very Fine Resolution Urban Scene Images

Transferable and Principled Efficiency for Open-Vocabulary Segmentation

Open-Vocabulary Semantic Segmentation with Decoupled One-Pass Network

Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation

SegViTv2: Exploring Efficient and Continual Semantic Segmentation with Plain Vision Transformers.

Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation

BAGNet: Branch Attention Guided Decoder for Semantic Segmentation