Abstract:Vision Transformers (ViTs) have emerged as the backbone of many segmentation models, consistently achieving state-of-the-art (SOTA) performance. However, their success comes at a significant computational cost. Image token pruning is one of the most effective strategies to address this complexity. However, previous approaches fall short when applied to more complex task-oriented segmentation (TOS), where the class of each image patch is not predefined but dependent on the specific input task. This work introduces the Vision Language Guided Token Pruning (VLTP), a novel token pruning mechanism that can accelerate ViTbased segmentation models, particularly for TOS guided by multi-modal large language model (MLLM). We argue that ViT does not need to process every image token through all of its layers only the tokens related to reasoning tasks are necessary. We design a new pruning decoder to take both image tokens and vision-language guidance as input to predict the relevance of each image token to the task. Only image tokens with high relevance are passed to deeper layers of the ViT. Experiments show that the VLTP framework reduces the computational costs of ViT by approximately 25% without performance degradation and by around 40% with only a 1% performance drop.

What problem does this paper attempt to address?

The paper aims to address the computational efficiency issues of Vision Transformers (ViTs) in Task-Oriented Segmentation (TOS). Specifically: 1. **Background and Challenges**: - ViTs perform excellently in image segmentation tasks, but their computational cost is high. - Token pruning is an effective strategy to improve efficiency, but it performs poorly in complex task-oriented segmentation because the category of each image token is not predefined and depends on the specific input task. 2. **Research Objectives**: - Propose a new Vision-Language Guided Token Pruning (VLTP) method to accelerate ViT-based segmentation models, especially in task-oriented segmentation guided by multimodal large language models (MLLM). - Ensure that the model only processes image tokens relevant to the inference task, thereby reducing unnecessary computation while maintaining high segmentation quality. 3. **Main Contributions**: - Designed a new pruning decoder that combines visual information and task reasoning to predict the relevance of each image token. - Experimental results show that the VLTP framework reduces computational cost by about 25% without performance degradation and by about 40% with only a 1% performance drop. - By integrating the powerful reasoning capabilities of MLLM, VLTP outperforms existing methods in multiple benchmarks, particularly in task-oriented segmentation tasks. In summary, this paper addresses the computational efficiency issues of ViT in complex task-oriented segmentation by introducing the VLTP method and demonstrates its significant advantages in practical applications.

VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation