Learning Transferable Visual Models From Natural Language Supervision, Feb. 2021

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever

Abstract:

What problem does this paper attempt to address?

Learning transferable visual models from natural language supervision. arXiv 2021

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever

2021-01-01

Abstract:
Learning transferable visual models from natural language supervision. arXiv

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever

2021-01-01

Abstract:
Learning Transferable Visual Models From Natural Language Supervision

Alec Radford,Jong Wook Kim,Chris Hallacy,Aditya Ramesh,Gabriel Goh,Sandhini Agarwal,Girish Sastry,Amanda Askell,Pamela Mishkin,Jack Clark,Gretchen Krueger,Ilya Sutskever

DOI: https://doi.org/10.48550/arXiv.2103.00020

2021-02-27

Abstract:State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at <a class="link-external link-https" href="https://github.com/OpenAI/CLIP" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Machine Learning
Knowledge Transfer Across Modalities with Natural Language Supervision

Carlo Alberto Barbano,Luca Molinaro,Emanuele Aiello,Marco Grangetto

2024-11-24

Abstract:We present a way to learn novel concepts by only using their textual description. We call this method Knowledge Transfer. Similarly to human perception, we leverage cross-modal interaction to introduce new concepts. We hypothesize that in a pre-trained visual encoder there are enough low-level features already learned (e.g. shape, appearance, color) that can be used to describe previously unknown high-level concepts. Provided with a textual description of the novel concept, our method works by aligning the known low-level features of the visual encoder to its high-level textual description. We show that Knowledge Transfer can successfully introduce novel concepts in multimodal models, in a very efficient manner, by only requiring a single description of the target concept. Our approach is compatible with both separate textual and visual encoders (e.g. CLIP) and shared parameters across modalities. We also show that, following the same principle, Knowledge Transfer can improve concepts already known by the model. Leveraging Knowledge Transfer we improve zero-shot performance across different tasks such as classification, segmentation, image-text retrieval, and captioning.

Computer Vision and Pattern Recognition
Neural Task Representations as Weak Supervision for Model Agnostic Cross-Lingual Transfer

Sujay Kumar Jauhar,Michael Gamon,Patrick Pantel

DOI: https://doi.org/10.48550/arXiv.1811.01115

2018-11-03

Abstract:Natural language processing is heavily Anglo-centric, while the demand for models that work in languages other than English is greater than ever. Yet, the task of transferring a model from one language to another can be expensive in terms of annotation costs, engineering time and effort. In this paper, we present a general framework for easily and effectively transferring neural models from English to other languages. The framework, which relies on task representations as a form of weak supervision, is model and task agnostic, meaning that many existing neural architectures can be ported to other languages with minimal effort. The only requirement is unlabeled parallel data, and a loss defined over task representations. We evaluate our framework by transferring an English sentiment classifier to three different languages. On a battery of tests, we show that our models outperform a number of strong baselines and rival state-of-the-art results, which rely on more complex approaches and significantly more resources and data. Additionally, we find that the framework proposed in this paper is able to capture semantically rich and meaningful representations across languages, despite the lack of direct supervision.

Computation and Language
A Visual Analytical Approach for Transfer Learning in Classification.

Yuxin Ma,Jiayi Xu,Xiangyang Wu,Fei Wang,Wei Chen

DOI: https://doi.org/10.1016/j.ins.2016.03.021

IF: 8.1

2017-01-01

Information Sciences

Abstract:Classification can be highly challenging when the dataset is extremely large, or when the training data in the underlying domain are difficult to obtain. One feasible solution to this challenge is transfer learning, which extracts the knowledge from source tasks and applies the knowledge to target tasks. Extant transfer learning schemes typically assume that similarities between the source task and the target task to some degree. This assumption does not hold in certain actual applications; analysts unfamiliar with the learning strategy can be frustrated by the complicated transfer relations and the non-intuitive transfer process. This paper presents a suite of visual communication and interaction techniques to support the transfer learning process. Furthermore, a pioneering visual-assisted transfer learning methodology is proposed in the context of classification. Our solution includes a visual communication interface that allows for comprehensive exploration of the entire knowledge transfer process and the relevance among tasks. With these techniques and the methodology, the analysts can intuitively choose relevant tasks and data, as well as iteratively incorporate their experience and expertise into the analysis process. We demonstrate the validity and efficiency of our visual design and the analysis approach with examples of text classification.
MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models

Masoud Monajatipoor,Liunian Harold Li,Mozhdeh Rouhsedaghat,Lin F. Yang,Kai-Wei Chang

2023-06-02

Abstract:Large-scale language models have shown the ability to adapt to a new task via conditioning on a few demonstrations (i.e., in-context learning). However, in the vision-language domain, most large-scale pre-trained vision-language (VL) models do not possess the ability to conduct in-context learning. How can we enable in-context learning for VL models? In this paper, we study an interesting hypothesis: can we transfer the in-context learning ability from the language domain to VL domain? Specifically, we first meta-trains a language model to perform in-context learning on NLP tasks (as in MetaICL); then we transfer this model to perform VL tasks by attaching a visual encoder. Our experiments suggest that indeed in-context learning ability can be transferred cross modalities: our model considerably improves the in-context learning capability on VL tasks and can even compensate for the size of the model significantly. On VQA, OK-VQA, and GQA, our method could outperform the baseline model while having 20 times fewer parameters.

Computation and Language
Transferring Vision-Language Models for Visual Recognition: A Classifier Perspective

Wenhao Wu,Zhun Sun,Yuxin Song,Jingdong Wang,Wanli Ouyang

DOI: https://doi.org/10.1007/s11263-023-01876-w

IF: 13.369

2023-09-09

International Journal of Computer Vision

Abstract:Transferring knowledge from pre-trained deep models for downstream tasks, particularly with limited labeled samples, is a fundamental problem in computer vision research. Recent advances in large-scale, task-agnostic vision-language pre-trained models, which are learned with billions of samples, have shed new light on this problem. In this study, we investigate how to efficiently transfer aligned visual and textual knowledge for downstream visual recognition tasks. We first revisit the role of the linear classifier in the vanilla transfer learning framework, and then propose a new paradigm where the parameters of the classifier are initialized with semantic targets from the textual encoder and remain fixed during optimization. To provide a comparison, we also initialize the classifier with knowledge from various resources. In the empirical study, we demonstrate that our paradigm improves the performance and training speed of transfer learning tasks. With only minor modifications, our approach proves effective across 17 visual datasets that span three different data domains: image, video, and 3D point cloud.

computer science, artificial intelligence
K-LITE: Learning Transferable Visual Models with External Knowledge

Sheng Shen,Chunyuan Li,Xiaowei Hu,Jianwei Yang,Yujia Xie,Pengchuan Zhang,Zhe Gan,Lijuan Wang,Lu Yuan,Ce Liu,Kurt Keutzer,Trevor Darrell,Anna Rohrbach,Jianfeng Gao

DOI: https://doi.org/10.48550/arXiv.2204.09222

2022-10-22

Abstract:The new generation of state-of-the-art computer vision systems are trained from natural language supervision, ranging from simple object category names to descriptive captions. This form of supervision ensures high generality and usability of the learned visual models, due to the broad concept coverage achieved via large-scale data collection process. Alternatively, we argue that learning with external knowledge is a promising way which leverages a much more structured source of supervision and offers sample efficiency. We propose K-LITE, a simple strategy to leverage external knowledge for building transferable visual systems: In training, it enriches entities in text with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that uses knowledge about the visual concepts. In evaluation, the text is also augmented with external knowledge and then used to reference learned visual concepts (or describe new ones) to enable zero-shot and few-shot transfer of the pre-trained models. We study the performance of K-LITE on two important computer vision problems, image classification and object detection, benchmarking on 20 and 13 different existing datasets, respectively. The proposed knowledge-augmented models show significant improvement in transfer learning performance over existing methods. Our code is available at <a class="link-external link-https" href="https://github.com/microsoft/klite" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence,Computation and Language,Machine Learning
Unsupervised Visual Attribute Transfer with Reconfigurable Generative Adversarial Networks

Taeksoo Kim,Byoungjip Kim,Moonsu Cha,Jiwon Kim

DOI: https://doi.org/10.48550/arXiv.1707.09798

2017-07-31

Abstract:Learning to transfer visual attributes requires supervision dataset. Corresponding images with varying attribute values with the same identity are required for learning the transfer function. This largely limits their applications, because capturing them is often a difficult task. To address the issue, we propose an unsupervised method to learn to transfer visual attribute. The proposed method can learn the transfer function without any corresponding images. Inspecting visualization results from various unsupervised attribute transfer tasks, we verify the effectiveness of the proposed method.

Computer Vision and Pattern Recognition
I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervision

Sophia Gu,Christopher Clark,Aniruddha Kembhavi

DOI: https://doi.org/10.48550/arXiv.2211.09778

2023-08-19

Abstract:Many high-level skills that are required for computer vision tasks, such as parsing questions, comparing and contrasting semantics, and writing descriptions, are also required in other domains such as natural language processing. In this paper, we ask whether it is possible to learn those skills from text data and then transfer them to vision tasks without ever training on visual training data. Key to our approach is exploiting the joint embedding space of contrastively trained vision and language encoders. In practice, there can be systematic differences between embedding spaces for different modalities in contrastive models, and we analyze how these differences affect our approach and study strategies to mitigate this concern. We produce models using only text training data on four representative tasks: image captioning, visual entailment, visual question answering and visual news captioning, and evaluate them on standard benchmarks using images. We find these models perform close to models trained on images, while surpassing prior work for captioning and visual entailment in this text-only setting by over 9 points, and outperforming all prior work on visual news by over 30 points. We also showcase a variety of stylistic image captioning models that are trained using no image data and no human-curated language data, but instead using readily-available text data from books, the web, or language models.

Computer Vision and Pattern Recognition,Computation and Language
Efficient Transfer Learning for Video-language Foundation Models

Haoxing Chen,Zizheng Huang,Yan Hong,Yanshuo Wang,Zhongcai Lyu,Zhuoer Xu,Jun Lan,Zhangxuan Gu

2024-11-18

Abstract:Pre-trained vision-language models provide a robust foundation for efficient transfer learning across various downstream tasks. In the field of video action recognition, mainstream approaches often introduce additional parameter modules to capture temporal information. While the increased model capacity brought by these additional parameters helps better fit the video-specific inductive biases, existing methods require learning a large number of parameters and are prone to catastrophic forgetting of the original generalizable knowledge. In this paper, we propose a simple yet effective Multi-modal Spatio-Temporal Adapter (MSTA) to improve the alignment between representations in the text and vision branches, achieving a balance between general knowledge and task-specific knowledge. Furthermore, to mitigate over-fitting and enhance generalizability, we introduce a spatio-temporal description-guided consistency constraint. This constraint involves feeding template inputs (i.e., ``a video of $\{\textbf{cls}\}$'') into the trainable language branch, while LLM-generated spatio-temporal descriptions are input into the pre-trained language branch, enforcing consistency between the outputs of the two branches. This mechanism prevents over-fitting to downstream tasks and improves the distinguishability of the trainable branch within the spatio-temporal semantic space. We evaluate the effectiveness of our approach across four tasks: zero-shot transfer, few-shot learning, base-to-novel generalization, and fully-supervised learning. Compared to many state-of-the-art methods, our MSTA achieves outstanding performance across all evaluations, while using only 2-7\% of the trainable parameters in the original model. Code will be avaliable at <a class="link-external link-https" href="https://github.com/chenhaoxing/ETL4Video" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
Understanding Visual Concepts Across Models

Brandon Trabucco,Max Gurinas,Kyle Doherty,Ruslan Salakhutdinov

2024-06-12

Abstract:Large multimodal models such as Stable Diffusion can generate, detect, and classify new visual concepts after fine-tuning just a single word embedding. Do models learn similar words for the same concepts (i.e. <orange-cat> = orange + cat)? We conduct a large-scale analysis on three state-of-the-art models in text-to-image generation, open-set object detection, and zero-shot classification, and find that new word embeddings are model-specific and non-transferable. Across 4,800 new embeddings trained for 40 diverse visual concepts on four standard datasets, we find perturbations within an $\epsilon$-ball to any prior embedding that generate, detect, and classify an arbitrary concept. When these new embeddings are spliced into new models, fine-tuning that targets the original model is lost. We show popular soft prompt-tuning approaches find these perturbative solutions when applied to visual concept learning tasks, and embeddings for visual concepts are not transferable. Code for reproducing our work is available at: <a class="link-external link-https" href="https://visual-words.github.io" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning
Leveraging Natural Supervision for Language Representation Learning and Generation

Mingda Chen

DOI: https://doi.org/10.48550/arXiv.2207.10617

2022-07-22

Abstract:Recent breakthroughs in Natural Language Processing (NLP) have been driven by language models trained on a massive amount of plain text. While powerful, deriving supervision from textual resources is still an open question. For example, language model pretraining often neglects the rich, freely-available structures in textual data. In this thesis, we describe three lines of work that seek to improve the training and evaluation of neural models using naturally-occurring supervision. We first investigate self-supervised training losses to help enhance the performance of pretrained language models for various NLP tasks. Specifically, we alter the sentence prediction loss to make it better suited to other pretraining losses and more challenging to solve. We design an intermediate finetuning step that uses self-supervised training to promote models' ability in cross-task generalization. Then we describe methods to leverage the structures in Wikipedia and paraphrases. In particular, we propose training losses to exploit hyperlinks, article structures, and article category graphs for entity-, discourse-, entailment-related knowledge. We propose a framework that uses paraphrase pairs to disentangle semantics and syntax in sentence representations. We extend the framework for a novel generation task that controls the syntax of output text with a sentential exemplar. Lastly, we discuss our work on tailoring textual resources for establishing challenging evaluation tasks. We introduce three datasets by defining novel tasks using various fan-contributed websites, including a long-form data-to-text generation dataset, a screenplay summarization dataset, and a long-form story generation dataset. These datasets have unique characteristics offering challenges to future work in their respective task settings.

Computation and Language
Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?

Tobias Norlund,Lovisa Hagström,Richard Johansson

DOI: https://doi.org/10.48550/arXiv.2109.11321

2021-09-30

Abstract:Large language models are known to suffer from the hallucination problem in that they are prone to output statements that are false or inconsistent, indicating a lack of knowledge. A proposed solution to this is to provide the model with additional data modalities that complements the knowledge obtained through text. We investigate the use of visual data to complement the knowledge of large language models by proposing a method for evaluating visual knowledge transfer to text for uni- or multimodal language models. The method is based on two steps, 1) a novel task querying for knowledge of memory colors, i.e. typical colors of well-known objects, and 2) filtering of model training data to clearly separate knowledge contributions. Additionally, we introduce a model architecture that involves a visual imagination step and evaluate it with our proposed method. We find that our method can successfully be used to measure visual knowledge transfer capabilities in models and that our novel model architecture shows promising results for leveraging multimodal knowledge in a unimodal setting.

Computation and Language
Exploring the Transferability of Visual Prompting for Multimodal Large Language Models

Yichi Zhang,Yinpeng Dong,Siyuan Zhang,Tianzan Min,Hang Su,Jun Zhu

2024-04-17

Abstract:Although Multimodal Large Language Models (MLLMs) have demonstrated promising versatile capabilities, their performance is still inferior to specialized models on downstream tasks, which makes adaptation necessary to enhance their utility. However, fine-tuning methods require independent training for every model, leading to huge computation and memory overheads. In this paper, we propose a novel setting where we aim to improve the performance of diverse MLLMs with a group of shared parameters optimized for a downstream task. To achieve this, we propose Transferable Visual Prompting (TVP), a simple and effective approach to generate visual prompts that can transfer to different models and improve their performance on downstream tasks after trained on only one model. We introduce two strategies to address the issue of cross-model feature corruption of existing visual prompting methods and enhance the transferability of the learned prompts, including 1) Feature Consistency Alignment: which imposes constraints to the prompted feature changes to maintain task-agnostic knowledge; 2) Task Semantics Enrichment: which encourages the prompted images to contain richer task-specific semantics with language guidance. We validate the effectiveness of TVP through extensive experiments with 6 modern MLLMs on a wide variety of tasks ranging from object recognition and counting to multimodal reasoning and hallucination correction.

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning
Unsupervised Prompt Learning for Vision-Language Models

Tony Huang,Jack Chu,Fangyun Wei

DOI: https://doi.org/10.48550/arXiv.2204.03649

2022-04-07

Computer Vision and Pattern Recognition

Abstract:Contrastive vision-language models like CLIP have shown great progress in transfer learning. In the inference stage, the proper text description, also known as prompt, needs to be carefully designed to correctly classify the given images. In order to avoid laborious prompt engineering, recent works such as CoOp, CLIP-Adapter and Tip-Adapter propose to adapt vision-language models for downstream image recognition tasks on a small set of labeled data. Though promising improvements are achieved, requiring labeled data from the target datasets may restrict the scalability. In this paper, we explore a different scenario, in which the labels of the target datasets are unprovided, and we present an unsupervised prompt learning (UPL) approach to avoid prompt engineering while simultaneously improving transfer performance of CLIP-like vision-language models. As far as we know, UPL is the first work to introduce unsupervised learning into prompt learning. Experimentally, our UPL outperforms original CLIP with prompt engineering on ImageNet as well as other 10 datasets. An enhanced version of UPL is even competitive with the 8-shot CoOp and the 8-shot TIP-Adapter on most datasets. Code and models are available at https://github.com/tonyhuang2022/UPL.
Image to Video Domain Adaptation Using Web Supervision

Andrew Kae,Yale Song

DOI: https://doi.org/10.48550/arXiv.1908.01449

2019-08-05

Abstract:Training deep neural networks typically requires large amounts of labeled data which may be scarce or expensive to obtain for a particular target domain. As an alternative, we can leverage webly-supervised data (i.e. results from a public search engine) which are relatively plentiful but may contain noisy results. In this work, we propose a novel two-stage approach to learn a video classifier using webly-supervised data. We argue that learning appearance features and then temporal features sequentially, rather than simultaneously, is an easier optimization for this task. We show this by first learning an image model from web images, which is used to initialize and train a video model. Our model applies domain adaptation to account for potential domain shift present between the source domain (webly-supervised data) and target domain and also accounts for noise by adding a novel attention component. We report results competitive with state-of-the-art for webly-supervised approaches on UCF-101 (while simplifying the training process) and also evaluate on Kinetics for comparison.

Computer Vision and Pattern Recognition
Failures to Find Transferable Image Jailbreaks Between Vision-Language Models

Rylan Schaeffer,Dan Valentine,Luke Bailey,James Chua,Cristóbal Eyzaguirre,Zane Durante,Joe Benton,Brando Miranda,Henry Sleight,John Hughes,Rajashree Agrawal,Mrinank Sharma,Scott Emmons,Sanmi Koyejo,Ethan Perez

2024-12-16

Abstract:The integration of new modalities into frontier AI systems offers exciting capabilities, but also increases the possibility such systems can be adversarially manipulated in undesirable ways. In this work, we focus on a popular class of vision-language models (VLMs) that generate text outputs conditioned on visual and textual inputs. We conducted a large-scale empirical study to assess the transferability of gradient-based universal image ``jailbreaks" using a diverse set of over 40 open-parameter VLMs, including 18 new VLMs that we publicly release. Overall, we find that transferable gradient-based image jailbreaks are extremely difficult to obtain. When an image jailbreak is optimized against a single VLM or against an ensemble of VLMs, the jailbreak successfully jailbreaks the attacked VLM(s), but exhibits little-to-no transfer to any other VLMs; transfer is not affected by whether the attacked and target VLMs possess matching vision backbones or language models, whether the language model underwent instruction-following and/or safety-alignment training, or many other factors. Only two settings display partially successful transfer: between identically-pretrained and identically-initialized VLMs with slightly different VLM training data, and between different training checkpoints of a single VLM. Leveraging these results, we then demonstrate that transfer can be significantly improved against a specific target VLM by attacking larger ensembles of ``highly-similar" VLMs. These results stand in stark contrast to existing evidence of universal and transferable text jailbreaks against language models and transferable adversarial attacks against image classifiers, suggesting that VLMs may be more robust to gradient-based transfer attacks.

Computation and Language,Artificial Intelligence,Cryptography and Security,Computer Vision and Pattern Recognition,Machine Learning

Learning Transferable Visual Models From Natural Language Supervision, Feb. 2021

Learning transferable visual models from natural language supervision. arXiv 2021

Learning transferable visual models from natural language supervision. arXiv

Learning Transferable Visual Models From Natural Language Supervision

Knowledge Transfer Across Modalities with Natural Language Supervision

Neural Task Representations as Weak Supervision for Model Agnostic Cross-Lingual Transfer

A Visual Analytical Approach for Transfer Learning in Classification.

MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models

Transferring Vision-Language Models for Visual Recognition: A Classifier Perspective

K-LITE: Learning Transferable Visual Models with External Knowledge

Unsupervised Visual Attribute Transfer with Reconfigurable Generative Adversarial Networks

I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervision

Efficient Transfer Learning for Video-language Foundation Models

Understanding Visual Concepts Across Models

Leveraging Natural Supervision for Language Representation Learning and Generation

Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?

Exploring the Transferability of Visual Prompting for Multimodal Large Language Models

Unsupervised Prompt Learning for Vision-Language Models

Image to Video Domain Adaptation Using Web Supervision

Failures to Find Transferable Image Jailbreaks Between Vision-Language Models