Abstract:Panoptic Part Segmentation (PPS) unifies panoptic and part segmentation into one task. Previous works utilize separate approaches to handle things, stuff, and part predictions without shared computation and task association. We aim to unify these tasks at the architectural level, designing the first end-to-end unified framework, Panoptic-PartFormer. Moreover, we find the previous metric PartPQ biases to PQ. To handle both issues, we first design a meta-architecture that decouples part features and things/stuff features, respectively. We model things, stuff, and parts as object queries and directly learn to optimize all three forms of prediction as a unified mask prediction and classification problem. We term our model as Panoptic-PartFormer. Second, we propose a new metric Part-Whole Quality (PWQ), better to measure this task from pixel-region and part-whole perspectives. It also decouples the errors for part segmentation and panoptic segmentation. Third, inspired by Mask2Former, based on our meta-architecture, we propose Panoptic-PartFormer++ and design a new part-whole cross-attention scheme to boost part segmentation qualities further. We design a new part-whole interaction method using masked cross attention. Finally, extensive ablation studies and analysis demonstrate the effectiveness of both Panoptic-PartFormer and Panoptic-PartFormer++. Compared with previous Panoptic-PartFormer, our Panoptic-PartFormer++ achieves 2% PartPQ and 3% PWQ improvements on the Cityscapes PPS dataset and 5% PartPQ on the Pascal Context PPS dataset. On both datasets, Panoptic-PartFormer++ achieves new state-of-the-art results. Our models can serve as a strong baseline and aid future research in PPS. The source code and trained models will be available at~\url{<a class="link-external link-https" href="https://github.com/lxtGH/Panoptic-PartFormer" rel="external noopener nofollow">this https URL</a>}.

SE-PSNet: Silhouette-based enhancement feature for panoptic segmentation network

An Improved Shared Encoder-Based Model for Fast Panoptic Segmentation

An End-to-End Network for Panoptic Segmentation

Dual-CNN Fusion Panoptic Segmentation Based on Edge Optimization

Unified Network With Detail Guidance for Panoptic Segmentation

Banet: Bidirectional Aggregation Network With Occlusion Handling For Panoptic Segmentation

Ensembling Instance and Semantic Segmentation for Panoptic Segmentation

Panoptic-PartFormer++: A Unified and Decoupled View for Panoptic Part Segmentation

Supervised Edge Attention Network for Accurate Image Instance Segmentation

Cascaded panoptic segmentation method for high resolution remote sensing image

Real-time Panoptic Segmentation with Relationship Between Adjacent Pixels and Boundary Prediction

Mask-Pyramid Network: A Novel Panoptic Segmentation Method

RefineMask: Towards High-Quality Instance Segmentation with Fine-Grained Features

BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation

PanopticPartFormer++: A Unified and Decoupled View for Panoptic Part Segmentation

Focusing on Flexible Masks: A Novel Framework for Panoptic Scene Graph Generation with Relation Constraints

Panoptic-PartFormer: Learning a Unified Model for Panoptic Part Segmentation

UPSNet: A Unified Panoptic Segmentation Network

SSPSNet: a Single Shot Panoptic Segmentation Network for Accurate Scene Parsing.

CASNet: Common Attribute Support Network for image instance and panoptic segmentation