Abstract:Panoptic Part Segmentation (PPS) aims to unify panoptic segmentation and part segmentation into one task. Previous work mainly utilizes separated approaches to handle thing, stuff, and part predictions individually without performing any shared computation and task association. In this work, we aim to unify these tasks at the architectural level, designing the first end-to-end unified method named Panoptic-PartFormer. In particular, motivated by the recent progress in Vision Transformer, we model things, stuff, and part as object queries and directly learn to optimize the all three predictions as unified mask prediction and classification problem. We design a decoupled decoder to generate part feature and thing/stuff feature respectively. Then we propose to utilize all the queries and corresponding features to perform reasoning jointly and iteratively. The final mask can be obtained via inner product between queries and the corresponding features. The extensive ablation studies and analysis prove the effectiveness of our framework. Our Panoptic-PartFormer achieves the new state-of-the-art results on both Cityscapes PPS and Pascal Context PPS datasets with at least 70% GFlops and 50% parameters decrease. In particular, we get 3.4% relative improvements with ResNet50 backbone and 10% improvements after adopting Swin Transformer on Pascal Context PPS dataset. To the best of our knowledge, we are the first to solve the PPS problem via \textit{a unified and end-to-end transformer model. Given its effectiveness and conceptual simplicity, we hope our Panoptic-PartFormer can serve as a good baseline and aid future unified research for PPS. Our code and models are available at <a class="link-external link-https" href="https://github.com/lxtGH/Panoptic-PartFormer" rel="external noopener nofollow">this https URL</a>.

Fully Convolutional Networks for Panoptic Segmentation

Omnisupervised Omnidirectional Semantic Segmentation

An End-to-End Network for Panoptic Segmentation

CenterLPS: Segment Instances by Centers for LiDAR Panoptic Segmentation

Deep Markov Clustering for Panoptic Segmentation.

An Improved Shared Encoder-Based Model for Fast Panoptic Segmentation

Panoptic Feature Pyramid Networks

Unified Network With Detail Guidance for Panoptic Segmentation

Depth-aware Panoptic Segmentation

Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentation

Panoptic Segmentation with a Joint Semantic and Instance Segmentation Network

Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers

MaskConver: Revisiting Pure Convolution Model for Panoptic Segmentation

UPSNet: A Unified Panoptic Segmentation Network

IDNet: Information Decomposition Network for Fast Panoptic Segmentation.

Attention-Guided Unified Network for Panoptic Segmentation

Unifying Instance and Panoptic Segmentation with Dynamic Rank-1 Convolutions

CFNet: Learning Correlation Functions for One-Stage Panoptic Segmentation

Cascaded panoptic segmentation method for high resolution remote sensing image

AinnoSeg: Panoramic Segmentation with High Perfomance

Panoptic-PartFormer: Learning a Unified Model for Panoptic Part Segmentation