Tag2Text: Guiding Vision-Language Model via Image Tagging
Xinyu Huang,Youcai Zhang,Jinyu Ma,Weiwei Tian,Rui Feng,Yuejie Zhang,Yaqian Li,Yandong Guo,Lei Zhang
DOI: https://doi.org/10.48550/arXiv.2303.05657
2024-03-18
Abstract:This paper presents Tag2Text, a vision language pre-training (VLP) framework, which introduces image tagging into vision-language models to guide the learning of visual-linguistic features. In contrast to prior works which utilize object tags either manually labeled or automatically detected with an off-the-shelf detector with limited performance, our approach explicitly learns an image tagger using tags parsed from image-paired text and thus provides a strong semantic guidance to vision-language models. In this way, Tag2Text can utilize large-scale annotation-free image tags in accordance with image-text pairs, and provides more diverse tag categories beyond objects. As a result, Tag2Text demonstrates the ability of a foundational image tagging model, with superior zero-shot performance even comparable to fully supervised models. Moreover, by leveraging the tagging guidance, Tag2Text effectively enhances the performance of vision-language models on both generation-based and alignment-based tasks. Across a wide range of downstream benchmarks, Tag2Text achieves state-of-the-art results with similar model sizes and data scales, demonstrating the efficacy of the proposed tagging guidance. Code, demo and pre-trained models are available at <a class="link-external link-https" href="https://github.com/xinyu1205/recognize-anything" rel="external noopener nofollow">this https URL</a>.
Computer Vision and Pattern Recognition