Abstract:In this work, we address the challenging task of few-shot and zero-shot 3D point cloud semantic segmentation. The success of few-shot semantic segmentation in 2D computer vision is mainly driven by the pre-training on large-scale datasets like imagenet. The feature extractor pre-trained on large-scale 2D datasets greatly helps the 2D few-shot learning. However, the development of 3D deep learning is hindered by the limited volume and instance modality of datasets due to the significant cost of 3D data collection and annotation. This results in less representative features and large intra-class feature variation for few-shot 3D point cloud segmentation. As a consequence, directly extending existing popular prototypical methods of 2D few-shot classification/segmentation into 3D point cloud segmentation won't work as well as in 2D domain. To address this issue, we propose a Query-Guided Prototype Adaption (QGPA) module to adapt the prototype from support point clouds feature space to query point clouds feature space. With such prototype adaption, we greatly alleviate the issue of large feature intra-class variation in point cloud and significantly improve the performance of few-shot 3D segmentation. Besides, to enhance the representation of prototypes, we introduce a Self-Reconstruction (SR) module that enables prototype to reconstruct the support mask as well as possible. Moreover, we further consider zero-shot 3D point cloud semantic segmentation where there is no support sample. To this end, we introduce category words as semantic information and propose a semantic-visual projection model to bridge the semantic and visual spaces. Our proposed method surpasses state-of-the-art algorithms by a considerable 7.90% and 14.82% under the 2-way 1-shot setting on S3DIS and ScanNet benchmarks, respectively.

Zero-Shot Point Cloud Segmentation by Semantic-Visual Aware Synthesis.

See More and Know More: Zero-shot Point Cloud Segmentation via Multi-modal Visual Data

Affinity3D: Propagating Instance-Level Semantic Affinity for Zero-Shot Point Cloud Semantic Segmentation

Weakly Supervised Classification Model for Zero‐shot Semantic Segmentation

Delving into Shape-aware Zero-shot Semantic Segmentation

Multi-modal Generative Adversarial Network for Zero-Shot Learning

Visual feature synthesis with semantic reconstructor for traditional and generalized zero‐shot object classification

A Meaningful Learning Method for Zero-Shot Semantic Segmentation

Generalized Zero Shot Learning Via Synthesis Pseudo Features.

Visual Data Synthesis Via GAN for Zero-Shot Video Classification

Primitive Generation and Semantic-related Alignment for Universal Zero-Shot Segmentation

Zero and Few Shot Learning with Semantic Feature Synthesis and Competitive Learning

From Pixel to Patch: Synthesize Context-Aware Features for Zero-Shot Semantic Segmentation

Zero-Shot Learning from Adversarial Feature Residual to Compact Visual Feature

Robust Region Feature Synthesizer for Zero-Shot Object Detection

Deep semantic-aware network for zero-shot visual urban perception

ZeroPS: High-quality Cross-modal Knowledge Transfer for Zero-Shot 3D Part Segmentation

Generalized Zero-Shot Recognition based on Visually Semantic Embedding

MeshSegmenter: Zero-Shot Mesh Semantic Segmentation via Texture Synthesis

Visual semantic segmentation based on few/zero-shot learning: An overview

Prototype Adaption and Projection for Few- and Zero-Shot 3D Point Cloud Semantic Segmentation