Abstract:3D visual grounding is a fundamental yet important task in multimedia understanding, which aims to locate a specific object in a complicated 3D scene semantically according to a text description. However, this task requires a large number of annotations of labeled text-object pairs for training, so the scarcity of annotated data has been a key obstacle in this task. To this end, this paper makes the first attempt to introduce and address a new semi-supervised setting, where only a few text-object labels are provided during training. Considering most scene data has no annotation, we explore a new solution for unlabeled 3D grounding by additionally training and transferring knowledge from a correlated task, i.e., 3D captioning. Our main insight is that 3D grounding and captioning are complementary and can be iteratively trained with unlabeled data to provide object and text contexts for each other with pseudo-label learning. Specifically, we propose a novel 3D Cross-Task Teacher-Student Framework (3D-CTTSF) for joint 3D grounding and captioning in the semi-supervised setting, where each branch contains parallel grounding and captioning modules. We first pre-train the two modules of the teacher branch with limited labeled data for warm-up. Then, we train the student branch to mimic the ability of the teacher model and iteratively update both branches with the unlabeled data. In particular, we transfer the learned knowledge between the grounding and captioning modules across two branches to generate and refine the pseudo-labels of unlabeled data for providing reliable supervision. To further improve the quality of the pseudo-labels, we design a cross-task pseudo-label generation scheme, filtering low-quality pseudo-labels at the detection, captioning, and grounding levels, respectively. Experimental results on various datasets show competitive performances in both tasks compared to previous fully- and weakly-supervised methods, demonstrating the proposed 3D-CTTSF can serve as an effective solution to overcome the data scarcity issue.

Cognition Transferring and Decoupling for Text-supervised Egocentric Semantic Segmentation

3D Object Segmentation Using Cross-Window Point Transformer with Latent Semantic Boundary Guidance

Adept: Annotation-denoising Auxiliary Tasks with Discrete Cosine Transform Map and Keypoint for Human-Centric Pretraining

Boosting Semantic Segmentation from the Perspective of Explicit Class Embeddings

Knowledge Transfer with Simulated Inter-Image Erasing for Weakly Supervised Semantic Segmentation

Conceptual Text Region Network: Cognition-Inspired Accurate Scene Text Detection

Open-Vocabulary Semantic Segmentation with Image Embedding Balancing

Cross-Task Knowledge Transfer for Semi-supervised Joint 3D Grounding and Captioning

Global Context Dependencies Aware Network for Efficient Semantic Segmentation of Fine-Resolution Remoted Sensing Images

Text2Seg: Remote Sensing Image Semantic Segmentation via Text-Guided Visual Foundation Models

Task-Specific Context Decoupling for Object Detection

Transformer-CNN Cohort: Semi-supervised Semantic Segmentation by the Best of Both Students

TIKP: Text-to-Image Knowledge Preservation for Continual Semantic Segmentation

Context-Aware Interaction Network for RGB-T Semantic Segmentation

TagCLIP: Improving Discrimination Ability of Open-Vocabulary Semantic Segmentation

Task Decomposition and Synchronization for Semantic Biomedical Image Segmentation

3D Open-Vocabulary Panoptic Segmentation with 2D-3D Vision-Language Distillation

Break the Bias: Delving Semantic Transform Invariance for Few-Shot Segmentation

DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model

DecoupleNet: Decoupled Network for Domain Adaptive Semantic Segmentation