Abstract:Panoptic Part Segmentation (PPS) aims to unify panoptic segmentation and part segmentation into one task. Previous work mainly utilizes separated approaches to handle thing, stuff, and part predictions individually without performing any shared computation and task association. In this work, we aim to unify these tasks at the architectural level, designing the first end-to-end unified method named Panoptic-PartFormer. In particular, motivated by the recent progress in Vision Transformer, we model things, stuff, and part as object queries and directly learn to optimize the all three predictions as unified mask prediction and classification problem. We design a decoupled decoder to generate part feature and thing/stuff feature respectively. Then we propose to utilize all the queries and corresponding features to perform reasoning jointly and iteratively. The final mask can be obtained via inner product between queries and the corresponding features. The extensive ablation studies and analysis prove the effectiveness of our framework. Our Panoptic-PartFormer achieves the new state-of-the-art results on both Cityscapes PPS and Pascal Context PPS datasets with at least 70% GFlops and 50% parameters decrease. In particular, we get 3.4% relative improvements with ResNet50 backbone and 10% improvements after adopting Swin Transformer on Pascal Context PPS dataset. To the best of our knowledge, we are the first to solve the PPS problem via \textit{a unified and end-to-end transformer model. Given its effectiveness and conceptual simplicity, we hope our Panoptic-PartFormer can serve as a good baseline and aid future unified research for PPS. Our code and models are available at <a class="link-external link-https" href="https://github.com/lxtGH/Panoptic-PartFormer" rel="external noopener nofollow">this https URL</a>.

Position-Guided Point Cloud Panoptic Segmentation Transformer

3D Object Segmentation Using Cross-Window Point Transformer with Latent Semantic Boundary Guidance

SEFormer: Structure Embedding Transformer for 3D Object Detection

Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers

MP-Former: Mask-Piloted Transformer for Image Segmentation

Stratified Transformer for 3D Point Cloud Segmentation

PVT-SSD: Single-Stage 3D Object Detector with Point-Voxel Transformer

SDPT: Semantic-Aware Dimension-Pooling Transformer for Image Segmentation

Mask4Former: Mask Transformer for 4D Panoptic Segmentation

OneFormer3D: One Transformer for Unified Point Cloud Segmentation

Panoptic-PartFormer: Learning a Unified Model for Panoptic Part Segmentation

PPTFormer: Pseudo Multi-Perspective Transformer for UAV Segmentation

PReFormer: A memory-efficient transformer for point cloud semantic segmentation

SEED: A Simple and Effective 3D DETR in Point Clouds

Dynamic Focus-aware Positional Queries for Semantic Segmentation

PCTrans: Position-Guided Transformer with Query Contrast for Biological Instance Segmentation

3DPPE: 3D Point Positional Encoding for Multi-Camera 3D Object Detection Transformers

3DPCT: 3D Point Cloud Transformer with Dual Self-attention

An Efficient 3-D Point Cloud Place Recognition Approach Based on Feature Point Extraction and Transformer

OctFormer: Octree-based Transformers for 3D Point Clouds

MECPformer: Multi-estimations Complementary Patch with CNN-Transformers for Weakly Supervised Semantic Segmentation